Python (boto) 启动 Spark 集群时出现类型错误

Question

以下是尝试启动具有 10 个从属节点的集群。

12:13:44/sparkup $ec2/spark-ec2 -k sparkeast -i ~/.ssh/myPem.pem \
          -s 10  -z us-east-1a -r us-east-1 launch spark2

这是输出。请注意，相同的命令已成功用于二月主代码。今天我已经更新到最新的 1.4.0-SNAPSHOT

Setting up security groups...
Searching for existing cluster spark2 in region us-east-1...
Spark AMI: ami-5bb18832
Launching instances...
Launched 10 slaves in us-east-1a, regid = r-68a0ae82
Launched master in us-east-1a, regid = r-6ea0ae84
Waiting for AWS to propagate instance metadata...
Waiting for cluster to enter 'ssh-ready' state.........unable to load cexceptions
TypeError
p0
(S''
p1
tp2
Rp3
(dp4
S'child_traceback'
p5
S'Traceback (most recent call last):\n  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1280, in _execute_child\n    sys.stderr.write("%s %s (env=%s)\n" %(executable, \' \'.join(args), \' \'.join(env)))\nTypeError\n'
p6
sb.Traceback (most recent call last):
  File "ec2/spark_ec2.py", line 1444, in <module>
    main()
  File "ec2/spark_ec2.py", line 1436, in main
    real_main()
  File "ec2/spark_ec2.py", line 1270, in real_main
    cluster_state='ssh-ready'
  File "ec2/spark_ec2.py", line 869, in wait_for_cluster_state
    is_cluster_ssh_available(cluster_instances, opts):
  File "ec2/spark_ec2.py", line 833, in is_cluster_ssh_available
    if not is_ssh_available(host=dns_name, opts=opts):
  File "ec2/spark_ec2.py", line 807, in is_ssh_available
    stderr=subprocess.STDOUT  # we pipe stderr through stdout to preserve output order
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1328, in _execute_child
    raise child_exception
TypeError

AWS 控制台显示实例实际上是运行。所以不清楚究竟是什么失败了。

任何提示或解决方法表示赞赏。

UPDATE 执行 login 命令时也会出现同样的错误。 boto 似乎有问题 API - 但集群本身似乎没问题。

ec2/spark-ec2  -i ~/.ssh/sparkeast.pem  login spark2
Searching for existing cluster spark2 in region us-east-1...
Found 1 master, 10 slaves.
Logging into master ec2-54-87-46-170.compute-1.amazonaws.com...
unable to load cexceptions
TypeError
p0
(..  same exception stacktrace as above )

Answer 1

问题是我的 yosemite macbook 上的 python-2.7.6 安装似乎已损坏。

我重置了 PATH 和 PYTHONPATH 以指向安装的自定义自制程序 python 版本，然后是 boto - 和其他 python 命令，包括构建 spark 性能项目 - 工作正常。

Python (boto) 启动 Spark 集群时出现类型错误

Python (boto) TypeError launching Spark Cluster

amazon-ec2

boto

apache-spark