EnvironmentError: Not in a PBS job

Question

我正在尝试使用以下代码在远程集群节点 (rh7) 上对一些脚本进行排队：

#!/bin/bash
#PBS -N spark-cluster-path
#PBS -l select=2:ncpus=4:mem=2G
#PBS -l walltime=01:00:00

# Qsub template
# Scheduler: PBS

#Environment
export JAVA_HOME=/some/path/to/jdk/1.8.0_112
export SPARK_HOME=/some/path/to/spark/2.2.1
export PYTHONPATH=/some/path/to/python3.6/dist-packages
export PYSPARK_PYTHON=/some/path/to/Python/3.6.5/bin/python3
export PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON
export PATH=$JAVA_HOME/bin:$SPARK_HOME/bin:$PYTHONPATH:$PATH

/some/path/to/spark/spark-on-pbs-0.1/pbs-spark-launcher ./word_count.py

变量都是正确的，脚本 word_count.py 独立工作得很好，但是当我运行这个脚本时，我得到以下错误：

File "/some/path/to/spark/spark-on-pbs-0.1/pbs-spark-launcher", line 118, in <module>
    raise EnvironmentError("Not in a PBS job")
EnvironmentError: Not in a PBS job

Google这里一片空白，我也是。这个问题从何而来？

Answer 1

引用 https://www.osc.edu/~troy/pbstools/man/pbs-spark-submit :

ASSUMPTIONS AND LIMITATIONS

pbs-spark-submit makes two assumptions about its environment. First, the Spark master process will be run on the PBS job's mother superior node. Second, the working directory for the Spark programs is on a file system shared across on nodes allocated to the job.

并且通过查看 code 看起来需要一个环境变量 PBS_JOBID

EnvironmentError: Not in a PBS job

EnvironmentError: Not in a PBS job

python

apache-spark

pbs