我无法在 EMR PySpark 笔记本中安装 spacy 模型
I can't install spacy model in EMR PySpark notebook
我目前有一个 AWS EMR,其中有一个链接到同一集群的笔记本。
我想加载一个 spacy 模型 (en_core_web_sm
) 但首先我需要下载通常使用 python -m spacy download en_core_web_sm
完成的模型,但我真的找不到如何在PySpark 会话。
这是我的配置:
%%configure -f
{
"name":"conf0",
"kind": "pyspark",
"conf":{
"spark.pyspark.python": "python",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/usr/bin/virtualenv"
},
"files":["s3://my-s3/code/utils/NLPtools.py",
"s3://my-s3/code/utils/Parse_wikidump.py",
"s3://my-s3/code/utils/S3_access.py",
"s3://my-s3/code/utils/myeval.py",
"s3://my-s3/code/utils/rank_metrics.py",
"s3://my-s3/code/utils/removeoutput.py",
"s3://my-s3/code/utils/secret_manager.py",
"s3://my-s3/code/utils/word2vec.py"]
}
我可以运行这样的命令,这很正常:
sc.install_pypi_package("boto3")
sc.install_pypi_package("pandas")
sc.install_pypi_package("hdfs")
sc.install_pypi_package("NLPtools")
sc.install_pypi_package("numpy")
sc.install_pypi_package("tqdm")
sc.install_pypi_package("wikipedia")
sc.install_pypi_package("filechunkio")
sc.install_pypi_package("thinc")
sc.install_pypi_package("gensim")
sc.install_pypi_package("termcolor")
sc.install_pypi_package("boto")
sc.install_pypi_package("spacy")
sc.install_pypi_package("langdetect")
sc.install_pypi_package("pathos")
但是当然,就像我无法成功下载模型一样,在尝试加载它时出现以下错误:
An error was encountered:
[E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
Traceback (most recent call last):
File "/mnt/tmp/spark-eef27750-07a4-4a8a-82dc-b006827e7f1f/userFiles-ec6ecbe3-558b-42df-bd38-cd33b2340ae0/NLPtools.py", line 13, in <module>
nlp = spacy.load('en_core_web_sm', disable=['parser', 'textcat'])
File "/tmp/1596550154785-0/lib/python2.7/site-packages/spacy/__init__.py", line 30, in load
return util.load_model(name, **overrides)
File "/tmp/1596550154785-0/lib/python2.7/site-packages/spacy/util.py", line 175, in load_model
raise IOError(Errors.E050.format(name=name))
IOError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
我尝试将它直接安装在集群上 (master/worker),但它位于 PySpark 会话“外部”,因此不会被嵌入。 !python -m spacy download en_core_web_sm
之类的命令在 PySpark 笔记本中不起作用...
提前致谢!
安装 spacy 和模型的最佳方法是使用 EMR bootsrap 脚本。这个对我有用。
我的配置:
Release label:emr-5.32.0
Hadoop distribution:Amazon 2.10.1
Applications:Spark 2.4.7
JupyterEnterpriseGateway 2.1.0
Livy 0.7.0
我的脚本:
#!/bin/bash -xe
#### WARNING #####
## After modifying this script you have to push it on s3
# Non-standard and non-Amazon Machine Image Python modules:
version=1.1
printf "This is the latest script $version"
sudo /usr/bin/pip3.7 install -U \
boto3 \
pandas \
langdetect \
hdfs \
tqdm \
pathos \
wikipedia \
filechunkio \
gensim \
termcolor \
awswrangler
# Install spacy. Order matter !
sudo /usr/bin/pip3.7 install -U \
numpy \
Cython \
pip
sudo /usr/local/bin/pip3.7 install -U spacy
sudo python3 -m spacy download en_core_web_sm
注意两点:
- 对所有命令使用 sudo
- 升级 pip 并更改路径
我目前有一个 AWS EMR,其中有一个链接到同一集群的笔记本。
我想加载一个 spacy 模型 (en_core_web_sm
) 但首先我需要下载通常使用 python -m spacy download en_core_web_sm
完成的模型,但我真的找不到如何在PySpark 会话。
这是我的配置:
%%configure -f
{
"name":"conf0",
"kind": "pyspark",
"conf":{
"spark.pyspark.python": "python",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/usr/bin/virtualenv"
},
"files":["s3://my-s3/code/utils/NLPtools.py",
"s3://my-s3/code/utils/Parse_wikidump.py",
"s3://my-s3/code/utils/S3_access.py",
"s3://my-s3/code/utils/myeval.py",
"s3://my-s3/code/utils/rank_metrics.py",
"s3://my-s3/code/utils/removeoutput.py",
"s3://my-s3/code/utils/secret_manager.py",
"s3://my-s3/code/utils/word2vec.py"]
}
我可以运行这样的命令,这很正常:
sc.install_pypi_package("boto3")
sc.install_pypi_package("pandas")
sc.install_pypi_package("hdfs")
sc.install_pypi_package("NLPtools")
sc.install_pypi_package("numpy")
sc.install_pypi_package("tqdm")
sc.install_pypi_package("wikipedia")
sc.install_pypi_package("filechunkio")
sc.install_pypi_package("thinc")
sc.install_pypi_package("gensim")
sc.install_pypi_package("termcolor")
sc.install_pypi_package("boto")
sc.install_pypi_package("spacy")
sc.install_pypi_package("langdetect")
sc.install_pypi_package("pathos")
但是当然,就像我无法成功下载模型一样,在尝试加载它时出现以下错误:
An error was encountered:
[E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
Traceback (most recent call last):
File "/mnt/tmp/spark-eef27750-07a4-4a8a-82dc-b006827e7f1f/userFiles-ec6ecbe3-558b-42df-bd38-cd33b2340ae0/NLPtools.py", line 13, in <module>
nlp = spacy.load('en_core_web_sm', disable=['parser', 'textcat'])
File "/tmp/1596550154785-0/lib/python2.7/site-packages/spacy/__init__.py", line 30, in load
return util.load_model(name, **overrides)
File "/tmp/1596550154785-0/lib/python2.7/site-packages/spacy/util.py", line 175, in load_model
raise IOError(Errors.E050.format(name=name))
IOError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
我尝试将它直接安装在集群上 (master/worker),但它位于 PySpark 会话“外部”,因此不会被嵌入。 !python -m spacy download en_core_web_sm
之类的命令在 PySpark 笔记本中不起作用...
提前致谢!
安装 spacy 和模型的最佳方法是使用 EMR bootsrap 脚本。这个对我有用。
我的配置:
Release label:emr-5.32.0
Hadoop distribution:Amazon 2.10.1
Applications:Spark 2.4.7
JupyterEnterpriseGateway 2.1.0
Livy 0.7.0
我的脚本:
#!/bin/bash -xe
#### WARNING #####
## After modifying this script you have to push it on s3
# Non-standard and non-Amazon Machine Image Python modules:
version=1.1
printf "This is the latest script $version"
sudo /usr/bin/pip3.7 install -U \
boto3 \
pandas \
langdetect \
hdfs \
tqdm \
pathos \
wikipedia \
filechunkio \
gensim \
termcolor \
awswrangler
# Install spacy. Order matter !
sudo /usr/bin/pip3.7 install -U \
numpy \
Cython \
pip
sudo /usr/local/bin/pip3.7 install -U spacy
sudo python3 -m spacy download en_core_web_sm
注意两点:
- 对所有命令使用 sudo
- 升级 pip 并更改路径