ImportError: cannot import name 'Pregel' from 'graphframes.lib'
ImportError: cannot import name 'Pregel' from 'graphframes.lib'
我正在使用来自 jupyter 的 pyspark 和 graphframes。我能够成功导入 pyspark 和 graphframes,但是当我尝试时:
from graphframes.lib import Pregel
我收到以下错误:
ImportError: cannot import name 'Pregel' from 'graphframes.lib'
这 post 是我如何让 graphframes 工作,但没有 graphframes.lib:
https://github.com/graphframes/graphframes/issues/104
wget https://github.com/graphframes/graphframes/archive/release-0.2.0.zip
unzip release-0.2.0.zip
cd graphframes-release-0.2.0
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-release-0.2.0/target/scala-2.11/graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar .
cp -r graphframes-release-0.2.0/python/graphframes .
# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# Launch the jupyter server.
pyspark --jars graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar
我尝试重复上述命令,没有环境线,因为 pyspark 在 jupyter 中对我来说工作得很好,使用不同的版本并且能够获得 graphframes.lib,但没有 Pregel:
wget https://github.com/graphframes/graphframes/archive/release-0.6.0.zip
unzip release-0.6.0.zip
cd graphframes-release-0.6.0
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-release-0.6.0/target/scala-2.11/graphframes-assembly-0.6.0-spark2.3.jar .
cp -r graphframes-release-0.6.0/python/graphframes .
# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# Launch the jupyter server.
pyspark --jars graphframes-assembly-0.6.0-spark2.3.jar
现在我可以看到 graphrames.lib 目录,但里面只有 aggregate_messages.py。
最后,我尝试了以下但得到了 404 错误:
wget https://github.com/graphframes/graphframes/archive/release-0.7.0.zip
我预计,因为我能够导入图形框架,所以我能够从 graphframes.lib 导入 Pregel。在我现在的 0.6.0 版本中,似乎有一个 graphrames.lib 但没有 Pregel,而且还没有 0.7.0 版本的 graphframes。
我能够使用以下方法解决此错误:
wget https://github.com/graphframes/graphframes/archive/f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
unzip graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
cd graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/target/scala-2.11/graphframes-assembly-0.7.0-spark2.4.jar .
cp -r graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/python/graphframes .
# Set environment to use Jupyter (if jupyter working with pyspark, skip)
# export PYSPARK_DRIVER_PYTHON=jupyter
# export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# launch pyspark
pyspark --jars graphframes-assembly-0.7.0-spark2.4.jar
我正在使用来自 jupyter 的 pyspark 和 graphframes。我能够成功导入 pyspark 和 graphframes,但是当我尝试时:
from graphframes.lib import Pregel
我收到以下错误:
ImportError: cannot import name 'Pregel' from 'graphframes.lib'
这 post 是我如何让 graphframes 工作,但没有 graphframes.lib:
https://github.com/graphframes/graphframes/issues/104
wget https://github.com/graphframes/graphframes/archive/release-0.2.0.zip
unzip release-0.2.0.zip
cd graphframes-release-0.2.0
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-release-0.2.0/target/scala-2.11/graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar .
cp -r graphframes-release-0.2.0/python/graphframes .
# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# Launch the jupyter server.
pyspark --jars graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar
我尝试重复上述命令,没有环境线,因为 pyspark 在 jupyter 中对我来说工作得很好,使用不同的版本并且能够获得 graphframes.lib,但没有 Pregel:
wget https://github.com/graphframes/graphframes/archive/release-0.6.0.zip
unzip release-0.6.0.zip
cd graphframes-release-0.6.0
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-release-0.6.0/target/scala-2.11/graphframes-assembly-0.6.0-spark2.3.jar .
cp -r graphframes-release-0.6.0/python/graphframes .
# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# Launch the jupyter server.
pyspark --jars graphframes-assembly-0.6.0-spark2.3.jar
现在我可以看到 graphrames.lib 目录,但里面只有 aggregate_messages.py。
最后,我尝试了以下但得到了 404 错误:
wget https://github.com/graphframes/graphframes/archive/release-0.7.0.zip
我预计,因为我能够导入图形框架,所以我能够从 graphframes.lib 导入 Pregel。在我现在的 0.6.0 版本中,似乎有一个 graphrames.lib 但没有 Pregel,而且还没有 0.7.0 版本的 graphframes。
我能够使用以下方法解决此错误:
wget https://github.com/graphframes/graphframes/archive/f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
unzip graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
cd graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/target/scala-2.11/graphframes-assembly-0.7.0-spark2.4.jar .
cp -r graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/python/graphframes .
# Set environment to use Jupyter (if jupyter working with pyspark, skip)
# export PYSPARK_DRIVER_PYTHON=jupyter
# export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# launch pyspark
pyspark --jars graphframes-assembly-0.7.0-spark2.4.jar