如何在 Hadoop 2.7.1 中获取作业计数器?
How do I get job counters in Hadoop 2.7.1?
我正在尝试使用 java 从特定作业中获取所有计数器...
我已经为 Hadoop 0.23.1 编写了工作代码:
JobClient client = new JobClient(new JobConf(createConfiguration()));
RunningJob job;
system.out.print("Looking for job with title containing the string '" + jobName + "'");
List<JobStatus> jobStatusList = Arrays.asList(client.getAllJobs());
Collections.sort(jobStatusList, new Comparator<JobStatus>() {
@Override
public int compare(JobStatus o1, JobStatus o2) {
return (o1.getStartTime() < o2.getStartTime()) ? 1 : (o1.getStartTime() > o2.getStartTime() ? -1 : 0);
}
});
for (JobStatus jobStatus : jobStatusList) {
job = client.getJob(jobStatus.getJobID());
if (job.getJobName().contains(jobName)) {
system.out.print(String.format("Job Name '%s' was found in job ID: %s...", jobName, jobStatus.getJobID().toString()));
return job.getCounters();
}
}
system.out.print("Did not find job that contains the string '" + jobName + "'. Counters are set to null.");
知道为什么它在 2.7.1 中不起作用吗?
REST 成功 API:
http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/counters
我正在尝试使用 java 从特定作业中获取所有计数器... 我已经为 Hadoop 0.23.1 编写了工作代码:
JobClient client = new JobClient(new JobConf(createConfiguration()));
RunningJob job;
system.out.print("Looking for job with title containing the string '" + jobName + "'");
List<JobStatus> jobStatusList = Arrays.asList(client.getAllJobs());
Collections.sort(jobStatusList, new Comparator<JobStatus>() {
@Override
public int compare(JobStatus o1, JobStatus o2) {
return (o1.getStartTime() < o2.getStartTime()) ? 1 : (o1.getStartTime() > o2.getStartTime() ? -1 : 0);
}
});
for (JobStatus jobStatus : jobStatusList) {
job = client.getJob(jobStatus.getJobID());
if (job.getJobName().contains(jobName)) {
system.out.print(String.format("Job Name '%s' was found in job ID: %s...", jobName, jobStatus.getJobID().toString()));
return job.getCounters();
}
}
system.out.print("Did not find job that contains the string '" + jobName + "'. Counters are set to null.");
知道为什么它在 2.7.1 中不起作用吗?
REST 成功 API:
http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/counters