Spring Data Flow Yarn - 无法访问 jarfile

Question

我尝试在 Spring Cloud Data Flow for Yarn 上运行简单 spring 批处理任务。不幸的是，在运行运行它时，我在 ResourceManager UI:

中收到错误消息

Application application_1473838120587_5156 failed 1 times due to AM    Container for appattempt_1473838120587_5156_000001 exited with exitCode:  1
For more detailed output, check application tracking page:http://ip-10-249-9-50.gc.stepstone.com:8088/cluster/app/application_1473838120587_5156Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1473838120587_5156_01_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.

来自 Appmaster.stderror 的更多信息指出：

Log Type: Appmaster.stderr
Log Upload Time: Mon Nov 07 12:59:57 +0000 2016
Log Length: 106
Error: Unable to access jarfile spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar

如果涉及到 Spring 云数据流，我正在尝试运行 in dataflow-shell:

app register --type task --name simple_batch_job --uri https://github.com/spring-cloud/spring-cloud-dataflow-samples/raw/master/tasks/simple-batch-job/batch-job-1.0.0.BUILD-SNAPSHOT.jar
task create foo --definition "simple_batch_job"
task launch foo

真的很难知道为什么会出现这个错误。我确定从数据流服务器到 yarn 的连接工作正常，因为在标准 HDFS 本地化 (/dataflow) 中复制了一些文件（servers.yml，带有作业和实用程序的 jar）但它在某种程度上无法访问。

我的 servers.yml 配置：

logging:
  level:
    org.apache.hadoop: DEBUG
    org.springframework.yarn: DEBUG
maven:
  remoteRepositories:
    springRepo:
      url: https://repo.spring.io/libs-snapshot
spring:
  main:
    show_banner: false
  hadoop:
    fsUri: hdfs://HOST:8020
    resourceManagerHost: HOST
    resourceManagerPort: 8032
    resourceManagerSchedulerAddress: HOST:8030
datasource:
    url: jdbc:h2:tcp://localhost:19092/mem:dataflow
    username: sa
    password:
    driverClassName: org.h2.Driver

我很高兴听到任何信息或 spring-yarn 提示和技巧来完成这项工作。

PS: 作为hadoop环境我使用Amazon EMR 5.0

编辑：来自 hdfs 的递归路径：

drwxrwxrwx   - user hadoop          0 2016-11-07 15:02 /dataflow/apps
drwxrwxrwx   - user hadoop          0 2016-11-07 15:02 /dataflow/apps/stream
drwxrwxrwx   - user hadoop          0 2016-11-07 15:04 /dataflow/apps/stream/app
-rwxrwxrwx   3 user hadoop        121 2016-11-07 15:05 /dataflow/apps/stream/app/application.properties
-rwxrwxrwx   3 user hadoop       1177 2016-11-07 15:04 /dataflow/apps/stream/app/servers.yml
-rwxrwxrwx   3 user hadoop   60202852 2016-11-07 15:04 /dataflow/apps/stream/app/spring-cloud-deployer-yarn-appdeployerappmaster-1.0.0.RELEASE.jar
drwxrwxrwx   - user hadoop          0 2016-11-04 14:22 /dataflow/apps/task
drwxrwxrwx   - user hadoop          0 2016-11-04 14:24 /dataflow/apps/task/app
-rwxrwxrwx   3 user hadoop        121 2016-11-04 14:25 /dataflow/apps/task/app/application.properties
-rwxrwxrwx   3 user hadoop       2101 2016-11-04 14:24 /dataflow/apps/task/app/servers.yml
-rwxrwxrwx   3 user hadoop   60198804 2016-11-04 14:24 /dataflow/apps/task/app/spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.RELEASE.jar
drwxrwxrwx   - user hadoop          0 2016-11-04 14:25 /dataflow/artifacts
drwxrwxrwx   - user hadoop          0 2016-11-07 15:06 /dataflow/artifacts/cache
-rwxrwxrwx   3 user hadoop   12323493 2016-11-04 14:25 /dataflow/artifacts/cache/https-c84ea9dc0103a4754aeb9a28bbc7a4f33c835854-batch-job-1.0.0.BUILD-SNAPSHOT.jar
-rwxrwxrwx   3 user hadoop   22139318 2016-11-07 15:07 /dataflow/artifacts/cache/log-sink-rabbit-1.0.0.BUILD-SNAPSHOT.jar
-rwxrwxrwx   3 user hadoop   12590921 2016-11-07 12:59 /dataflow/artifacts/cache/timestamp-task-1.0.0.BUILD-SNAPSHOT.jar

Answer 1

显然混合了错误的版本，因为 hdfs 有 spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.RELEASE.jar 并且错误抱怨 spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar。

除非手动构建分发，否则不确定如何获得快照？

我建议从 http://cloud.spring.io/spring-cloud-dataflow-server-yarn 中选择 1.0.2。请参阅参考文档中的 "Download and Extract Distribution"。同时从 hdfs 中删除旧的 /dataflow 目录。

Spring Data Flow Yarn - 无法访问 jarfile

Spring Data Flow Yarn - unable access jarfile

spring-batch

hadoop-yarn

spring-cloud

spring-cloud-task

spring-cloud-dataflow