Pyspark 一般导入问题

Question

我在我的机器上成功安装了 Spark 和 Pyspark，添加了路径变量等，但一直面临导入问题。

这是代码：

from pyspark.sql import SparkSession

spark = SparkSession.builder \
                    .config("spark.hadoop.hive.exec.dynamic.partition", "true") \
                    .config("spark.hadoop.hive.exec.dynamic.partition.mode", "nonstrict") \
                    .enableHiveSupport() \
                    .getOrCreate()

这是错误信息：

"C:\...\Desktop\Clube\venv\Scripts\python.exe" "C:.../Desktop/Clube/services/ce_modelo_analise.py"
Traceback (most recent call last):
  File "C:\...\Desktop\Clube\services\ce_modelo_analise.py", line 1, in <module>
    from pyspark.sql import SparkSession
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\context.py", line 31, in <module>
    from pyspark import accumulators
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "C:\Spark\spark-2.4.0-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(

TypeError: 'bytes' object cannot be interpreted as an integer

如果我删除导入行，那些问题就会消失。正如我之前所说，我的路径变量设置为：

和

此外，Spark 在 cmd 中运行正确：

Answer 1

更深入我发现了问题：我在 2.4 版中使用 Spark，它适用于 Python 3.7 顶。

当我使用 Python 3.10 时，出现了问题。

因此，如果您遇到同样的问题，请尝试更改您的版本。

Pyspark 一般导入问题

Pyspark general import problems

installation

path

apache-spark

pyspark