如何知道 Yarn 公平份额调度程序是否发生抢占？

Question

有什么方法可以确定YARN是否触发了抢占机制？

可能在 YARN 资源管理器或日志中？

Answer 1

如果您的日志级别设置为 info，您应该会在 YARN 资源管理器日志中看到它。

            // Warn application about containers to be killed
            for (RMContainer container : containers) {
              FSAppAttempt app = scheduler.getSchedulerApp(
                      container.getApplicationAttemptId());
              LOG.info("Preempting container " + container +
                      " from queue " + app.getQueueName());
              app.trackContainerForPreemption(container);
            }

在 https://github.com/apache/hadoop/.../scheduler/fair/FSPreemptionThread.java#L143。

如果您的日志级别为 debug，您还会看到：

    if (LOG.isDebugEnabled()) {
      LOG.debug(
          "allocate: post-update" + " applicationAttemptId=" + appAttemptId
              + " #ask=" + ask.size() + " reservation= " + application
              .getCurrentReservation());

      LOG.debug("Preempting " + preemptionContainerIds.size()
          + " container(s)");
    }

在 https://github.com/apache/hadoop/.../scheduler/fair/FairScheduler.java#937

如何知道 Yarn 公平份额调度程序是否发生抢占？

How to know if preemption is happening on Yarn fair share scheduler?

hadoop

hive

hadoop-yarn