java.lang.UnsatisfiedLinkError: jep.Jep.init(Ljava/lang/ClassLoader;ZZ) on google cloud dataproc
java.lang.UnsatisfiedLinkError: jep.Jep.init(Ljava/lang/ClassLoader;ZZ) on google cloud dataproc
首先我不明白为什么人们对这个问题打负分。要么解释我如何改进问题。我可以进一步详细说明。这是我这边的反馈。虽然我是新人，但我不打算不努力地提出问题。
我正在尝试 运行 在使用 jep 解释器的 Google Cloud Platform Dataproc 集群上用 Scala 编写 spark 作业。
我添加了 jep 作为依赖项。
使用 Google Cloud Platform Dataproc
运行 Scala 上的 jep 的完整简短解决方案是什么
"black.ninia" % "jep" % "3.9.0"
在我写的 install.sh 脚本中
sudo -E pip install jep
export JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
我仍然收到以下错误（java.library.path 中没有 jep）
20/01/07 09:07:23 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 4.0 in stage 9.0 (TID 74, fs-xxxx-xxx-xxxx-test-w-1.c.xx-xxxx.internal, executor 1): java.lang.UnsatisfiedLinkError: no jep in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at jep.MainInterpreter.initialize(MainInterpreter.java:128)
at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
at jep.Jep.<init>(Jep.java:256)
at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)
at dunnhumby.sciencebank.SubsCommons$$anonfun$getUnitVecEmbeddings.apply(SubsCommons.scala:33)
at dunnhumby.sciencebank.SubsCommons$$anonfun$getUnitVecEmbeddings.apply(SubsCommons.scala:31)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun.apply(objects.scala:196)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun.apply(objects.scala:193)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$anonfun$apply.apply(RDD.scala:827)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$anonfun$apply.apply(RDD.scala:827)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
（已编辑）：-
1.) 我看到了针对本地机器的具体答案，但没有看到针对 Google 云平台的具体答案。
2.) 我找到了 https://github.com/ninia/jep/issues/141 但这没有帮助
3.) 我也找到了 answer 但没有得到回答，Google 云平台也不接受。我什至从那里执行了所有步骤。
4.) 如果问题缺少一些快照，我会附上。但请提供一些意见。
（编辑：- 08012020 我正在添加 install.sh 使用）
#!/bin/bash
set -x -e
# Disable ipv6 since it seems to cause intermittent SocketTimeoutException when collecting data
# See CENG-1268 in Jira
printf "\nnet.ipv6.conf.default.disable_ipv6=1\nnet.ipv6.conf.all.disable_ipv6=1\n" >> /etc/sysctl.conf
sysctl -p
if [[ $(/usr/share/google/get_metadata_value attributes/dataproc-role) == Master ]]; then
config_bucket="$(/usr/share/google/get_metadata_value attributes/dataproc-cluster-configuration-directory | cut -d'/' -f3)"
dataproc_cluster_name="$(/usr/share/google/get_metadata_value attributes/dataproc-cluster-name)"
hdfs dfs -mkdir -p gs://${config_bucket}/${dataproc_cluster_name}/spark_events
systemctl restart spark-history-server.service
fi
tee -a /etc/hosts << EOM
$$(/usr/share/google/get_metadata_value /attributes/preprod-mjr-dataplatform-metrics-mig-ip) influxdb
EOM
echo "[global]
index-url = https://cs-anonymous:XXXXXXXX@artifactory.xxxxxxxx.com/artifactory/api/pypi/pypi-remote/simple" >/etc/pip.conf
PIP_REQUIREMENTS_FILE=gs://preprod-xxx-dpl-artif/dataproc/requirements.txt
PIP_TRANSITIVE_REQUIREMENTS_FILE=gs://preprod-xxx-dpl-artif/dataproc/transitive-requirements.txt
gsutil cp ${PIP_REQUIREMENTS_FILE} .
gsutil cp ${PIP_TRANSITIVE_REQUIREMENTS_FILE} .
gsutil -q cp gs://preprod-xxx-dpl-artif/dataproc/apt-transport-https_1.4.8_amd64.deb /tmp/apt-transport-https_1.4.8_amd64.deb
export http_proxy=http://preprod-xxx-securecomms.preprod-xxx-securecomms.il4.us-east1.lb.dh-xxxxx-media-55595.internal:3128
export https_proxy=http://preprod-xxx-securecomms.preprod-xxx-securecomms.il4.us-east1.lb.dh-xxxxx-media-55595.internal:3128
export no_proxy=google.com,googleapis.com,localhost
echo "deb https://cs-anonymous:Welcome123@artifactory.xxxxxxxx.com/artifactory/debian-main-remote stretch main" >/etc/apt/sources.list.d/main.list
echo "deb https://cs-anonymous:Welcome123@artifactory.xxxxxxxx.com/artifactory/maria-db-debian stretch main" >>/etc/apt/sources.list.d/main.list
echo 'Acquire::CompressionTypes::Order:: "gz";' > /etc/apt/apt.conf.d/02update
echo 'Acquire::http::Timeout "10";' > /etc/apt/apt.conf.d/99timeout
echo 'Acquire::ftp::Timeout "10";' >> /etc/apt/apt.conf.d/99timeout
sudo dpkg -i /tmp/apt-transport-https_1.4.8_amd64.deb
sudo apt-get install --allow-unauthenticated -y /tmp/apt-transport-https_1.4.8_amd64.deb
sudo -E apt-get update --allow-unauthenticated -y -o Dir::Etc::sourcelist="sources.list.d/main.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
sudo -E apt-get --allow-unauthenticated -y install python-pip gcc python-dev python-tk curl
#requires index-url specifying because the version of pip installed by previous command
#installs an old version that doesn't seem to recognise pip.conf
sudo -E pip install --index-url https://cs-anonymous:xxxxxxx@artifactory.xxxxxxxx.com/artifactory/api/pypi/pypi-remote/simple --ignore-installed pip setuptools wheel
sudo -E pip install jep
sudo -E pip install gensim
JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
cat << EOF >> /etc/spark/conf/spark-env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOF
tee -a /etc/spark/conf/spark-defaults.conf << EOM
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOM
tee -a /etc/*bashrc << EOM
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOM
source /etc/*bashrc
sudo -E apt-get install --allow-unauthenticated -y \
pkg-config \
freetype* \
python-matplotlib \
libpq-dev \
libssl-dev \
libcrypto* \
python-dev \
libtext-csv-xs-perl \
libmysqlclient-dev \
libfreetype* \
libzmq3-dev \
libzmq3*
sudo -E pip install -r ./requirements.txt
假设您使用 install.sh 作为 Dataproc 的初始化操作，您的 export 命令只会在本地 shell 会话 运行ning 中导出这些环境变量init 操作，对于之后 运行 的所有 Spark 进程不是持久的。
让Spark使用自定义环境变量的方法是将它们添加到/etc/spark/conf/spark-env.sh。这是 spark user discussion about how to set java.library.path in Spark.
基本上，您可以在围绕导出环境变量的部分的 init 操作中使用 heredoc。但是，如 https://issues.apache.org/jira/browse/SPARK-1719 the environment variable won't be sufficient to propagate the library path into the executors in YARN; spark explicitly sets the library path 所示，而不是通过 LD_LIBRARY_PATH 传播，因此我们必须在 spark-defaults.conf
中也使用 spark.executor.extraLibraryPath
JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
# spark-env.sh for driver process.
cat << EOF >> /etc/spark/conf/spark-env.sh
# Note that backslash before $LD_LIBRARY_PATH on the right hand side;
# it is important that the variable is evaluated in spark-env.sh rather
# than clobbering it with the local $LD_LIBRARY_PATH of the init action
# running process.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
EOF
# For executor processes
cat << EOF >> /etc/spark/conf/spark-defaults.conf
spark.executor.extraLibraryPath=$JEP_PATH/jep
EOF
首先我不明白为什么人们对这个问题打负分。要么解释我如何改进问题。我可以进一步详细说明。这是我这边的反馈。虽然我是新人，但我不打算不努力地提出问题。
我正在尝试 运行 在使用 jep 解释器的 Google Cloud Platform Dataproc 集群上用 Scala 编写 spark 作业。
我添加了 jep 作为依赖项。
使用 Google Cloud Platform Dataproc
运行 Scala 上的 jep 的完整简短解决方案是什么"black.ninia" % "jep" % "3.9.0"
在我写的 install.sh 脚本中
sudo -E pip install jep
export JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
我仍然收到以下错误（java.library.path 中没有 jep）
20/01/07 09:07:23 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 4.0 in stage 9.0 (TID 74, fs-xxxx-xxx-xxxx-test-w-1.c.xx-xxxx.internal, executor 1): java.lang.UnsatisfiedLinkError: no jep in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at jep.MainInterpreter.initialize(MainInterpreter.java:128)
at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
at jep.Jep.<init>(Jep.java:256)
at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)
at dunnhumby.sciencebank.SubsCommons$$anonfun$getUnitVecEmbeddings.apply(SubsCommons.scala:33)
at dunnhumby.sciencebank.SubsCommons$$anonfun$getUnitVecEmbeddings.apply(SubsCommons.scala:31)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun.apply(objects.scala:196)
at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun.apply(objects.scala:193)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$anonfun$apply.apply(RDD.scala:827)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$anonfun$apply.apply(RDD.scala:827)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
（已编辑）：-
1.) 我看到了针对本地机器的具体答案，但没有看到针对 Google 云平台的具体答案。
2.) 我找到了 https://github.com/ninia/jep/issues/141 但这没有帮助
3.) 我也找到了 answer 但没有得到回答，Google 云平台也不接受。我什至从那里执行了所有步骤。
4.) 如果问题缺少一些快照，我会附上。但请提供一些意见。
（编辑：- 08012020 我正在添加 install.sh 使用）
#!/bin/bash
set -x -e
# Disable ipv6 since it seems to cause intermittent SocketTimeoutException when collecting data
# See CENG-1268 in Jira
printf "\nnet.ipv6.conf.default.disable_ipv6=1\nnet.ipv6.conf.all.disable_ipv6=1\n" >> /etc/sysctl.conf
sysctl -p
if [[ $(/usr/share/google/get_metadata_value attributes/dataproc-role) == Master ]]; then
config_bucket="$(/usr/share/google/get_metadata_value attributes/dataproc-cluster-configuration-directory | cut -d'/' -f3)"
dataproc_cluster_name="$(/usr/share/google/get_metadata_value attributes/dataproc-cluster-name)"
hdfs dfs -mkdir -p gs://${config_bucket}/${dataproc_cluster_name}/spark_events
systemctl restart spark-history-server.service
fi
tee -a /etc/hosts << EOM
$$(/usr/share/google/get_metadata_value /attributes/preprod-mjr-dataplatform-metrics-mig-ip) influxdb
EOM
echo "[global]
index-url = https://cs-anonymous:XXXXXXXX@artifactory.xxxxxxxx.com/artifactory/api/pypi/pypi-remote/simple" >/etc/pip.conf
PIP_REQUIREMENTS_FILE=gs://preprod-xxx-dpl-artif/dataproc/requirements.txt
PIP_TRANSITIVE_REQUIREMENTS_FILE=gs://preprod-xxx-dpl-artif/dataproc/transitive-requirements.txt
gsutil cp ${PIP_REQUIREMENTS_FILE} .
gsutil cp ${PIP_TRANSITIVE_REQUIREMENTS_FILE} .
gsutil -q cp gs://preprod-xxx-dpl-artif/dataproc/apt-transport-https_1.4.8_amd64.deb /tmp/apt-transport-https_1.4.8_amd64.deb
export http_proxy=http://preprod-xxx-securecomms.preprod-xxx-securecomms.il4.us-east1.lb.dh-xxxxx-media-55595.internal:3128
export https_proxy=http://preprod-xxx-securecomms.preprod-xxx-securecomms.il4.us-east1.lb.dh-xxxxx-media-55595.internal:3128
export no_proxy=google.com,googleapis.com,localhost
echo "deb https://cs-anonymous:Welcome123@artifactory.xxxxxxxx.com/artifactory/debian-main-remote stretch main" >/etc/apt/sources.list.d/main.list
echo "deb https://cs-anonymous:Welcome123@artifactory.xxxxxxxx.com/artifactory/maria-db-debian stretch main" >>/etc/apt/sources.list.d/main.list
echo 'Acquire::CompressionTypes::Order:: "gz";' > /etc/apt/apt.conf.d/02update
echo 'Acquire::http::Timeout "10";' > /etc/apt/apt.conf.d/99timeout
echo 'Acquire::ftp::Timeout "10";' >> /etc/apt/apt.conf.d/99timeout
sudo dpkg -i /tmp/apt-transport-https_1.4.8_amd64.deb
sudo apt-get install --allow-unauthenticated -y /tmp/apt-transport-https_1.4.8_amd64.deb
sudo -E apt-get update --allow-unauthenticated -y -o Dir::Etc::sourcelist="sources.list.d/main.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
sudo -E apt-get --allow-unauthenticated -y install python-pip gcc python-dev python-tk curl
#requires index-url specifying because the version of pip installed by previous command
#installs an old version that doesn't seem to recognise pip.conf
sudo -E pip install --index-url https://cs-anonymous:xxxxxxx@artifactory.xxxxxxxx.com/artifactory/api/pypi/pypi-remote/simple --ignore-installed pip setuptools wheel
sudo -E pip install jep
sudo -E pip install gensim
JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
cat << EOF >> /etc/spark/conf/spark-env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOF
tee -a /etc/spark/conf/spark-defaults.conf << EOM
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOM
tee -a /etc/*bashrc << EOM
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
export LD_PRELOAD=$LD_PRELOAD:$JEP_PATH/jep
EOM
source /etc/*bashrc
sudo -E apt-get install --allow-unauthenticated -y \
pkg-config \
freetype* \
python-matplotlib \
libpq-dev \
libssl-dev \
libcrypto* \
python-dev \
libtext-csv-xs-perl \
libmysqlclient-dev \
libfreetype* \
libzmq3-dev \
libzmq3*
sudo -E pip install -r ./requirements.txt
假设您使用 install.sh 作为 Dataproc 的初始化操作，您的 export 命令只会在本地 shell 会话 运行ning 中导出这些环境变量init 操作，对于之后 运行 的所有 Spark 进程不是持久的。
让Spark使用自定义环境变量的方法是将它们添加到/etc/spark/conf/spark-env.sh。这是 spark user discussion about how to set java.library.path in Spark.
基本上，您可以在围绕导出环境变量的部分的 init 操作中使用 heredoc。但是，如 https://issues.apache.org/jira/browse/SPARK-1719 the environment variable won't be sufficient to propagate the library path into the executors in YARN; spark explicitly sets the library path 所示，而不是通过 LD_LIBRARY_PATH 传播，因此我们必须在 spark-defaults.conf
spark.executor.extraLibraryPath
JEP_PATH=$(pip show jep | grep "^Location:" | cut -d ':' -f 2,3 | cut -d ' ' -f 2)
# spark-env.sh for driver process.
cat << EOF >> /etc/spark/conf/spark-env.sh
# Note that backslash before $LD_LIBRARY_PATH on the right hand side;
# it is important that the variable is evaluated in spark-env.sh rather
# than clobbering it with the local $LD_LIBRARY_PATH of the init action
# running process.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JEP_PATH/jep
EOF
# For executor processes
cat << EOF >> /etc/spark/conf/spark-defaults.conf
spark.executor.extraLibraryPath=$JEP_PATH/jep
EOF