创建 step spark python，亚马逊 hadoop

Question

我正在 Amazon 上使用 Hadoop 创建一个 Spark 步骤，但我一直在思考。不是因为我代码不好或者发送错误的判断，而是找不到出路。

我传码

spark-submit --deploy-mode cluster --master yarn --num-executors 5 --executor-cores 5 --executor-memory 1g s3://URL-S3/scripts/test.py

脚本：

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('TestSpark')

table.put_item(
   Item={
        'app_token': "1a",
        'advertising_id': "1b",
    }
)

我一直回来

16/08/25 07:06:22 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:23 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:24 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:25 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:26 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:27 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:28 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:29 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:30 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:31 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:32 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:33 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:34 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:35 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:36 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:37 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:38 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:39 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:40 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:41 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)
16/08/25 07:06:42 INFO Client: Application report for application_1472106590712_0002 (state: ACCEPTED)

错误日志：

2016-08-25T07:30:14.769Z INFO Step created jobs: 
2016-08-25T07:30:14.769Z WARN Step failed with exitCode 1 and took 1062 seconds

谢谢！

这已经是错误了，但是模块和安装之前。

ImportError: No module named boto3

Answer 1

您的应用程序正在等待 yarn 资源。转到资源管理器 URL 并查看您是否有足够的资源并使用正确的队列。看看yarn resourcemanager的日志就知道原因了。

Answer 2

我不在 Amazon EMR 上工作，但在 Hadoop 中，当您的 YARN 等待资源的时间过长时，就会发生这种情况。

资源协商器无法分配所需的资源，请尝试减少您的代码所需的资源。还要检查日志。

通读：this

同时检查 YARN 的状态，

sudo service hadoop-yarn-nodemanager status
sudo service hadoop-yarn-resourcemanager status

Answer 3

并定位错误。

Boto3 模块未安装，从控制台安装它，但这些步骤不起作用，因为他们必须在所有实例中安装它。所以我所做的是创建另一个 claster 运行一个 boostrap-action 更新 python 我安装了模块 boto3

创建 step spark python，亚马逊 hadoop

create step spark python, amazon hadoop

python

hadoop

hive

pyspark

amazon-emr