Spark-Submit:无法使用 virtualenv 运行
Spark-Submit : Cannot run with virtualenv
我有一个 python 应用程序,我想 运行 使用 spark 提交通过虚拟环境。这是我的命令
PYSPARK_PYTHON=./venv/bin/python spark-submit --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv/bin/python --master yarn --deploy-mode cluster --archives venv.zip#venv test.py
这里,venv.zip
是存档的虚拟环境。现在,当我 运行 spark-submit 命令时,我在控制台上得到了这个
20/01/28 17:08:12 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at myMasterNode/some.ip:somePort
20/01/28 17:08:13 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at myMasterNode/some.ip:somePort
20/01/28 17:08:16 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1580155727514_5620
Exception in thread "main" org.apache.spark.SparkException: Application application_1580155727514_5620 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
在纱线日志上,这是我看到的
20/01/28 17:08:53 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program "./signal/bin/python": error=2, No such file or directory
java.io.IOException: Cannot run program "./venv/bin/python": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
我做错了什么?我怎样才能确保 venv.zip
被正确复制和解压缩?
解压后在 #venv 中放置了一个 venv 目录。
spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv/venv/bin/python
也应该如此
如果将 zip 更改为 tar.gz,此问题将消失
我有一个 python 应用程序,我想 运行 使用 spark 提交通过虚拟环境。这是我的命令
PYSPARK_PYTHON=./venv/bin/python spark-submit --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv/bin/python --master yarn --deploy-mode cluster --archives venv.zip#venv test.py
这里,venv.zip
是存档的虚拟环境。现在,当我 运行 spark-submit 命令时,我在控制台上得到了这个
20/01/28 17:08:12 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at myMasterNode/some.ip:somePort
20/01/28 17:08:13 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at myMasterNode/some.ip:somePort
20/01/28 17:08:16 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1580155727514_5620
Exception in thread "main" org.apache.spark.SparkException: Application application_1580155727514_5620 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
在纱线日志上,这是我看到的
20/01/28 17:08:53 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program "./signal/bin/python": error=2, No such file or directory
java.io.IOException: Cannot run program "./venv/bin/python": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
我做错了什么?我怎样才能确保 venv.zip
被正确复制和解压缩?
解压后在 #venv 中放置了一个 venv 目录。
spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv/venv/bin/python
也应该如此如果将 zip 更改为 tar.gz,此问题将消失