尝试使用 pyspark 访问 greenplum table 时出错
Errors when trying to access greenplum table using pyspark
我正在尝试登录服务器并提取数据以在 python 中本地构建模型。我正在使用 pyspark 库 - 但我不断收到相同的错误。这是我 运行ning 的代码，没有错误，所以我知道我已经正确安装了 .jar 文件：
import pyspark
print(pyspark.__file__)
from pyspark.sql import DataFrameReader
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext(appName="PythonStreamingQueueStream")
sqlContext = SQLContext(sc)
sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider')
sc.stop()
它提供此作为指示 jar 已正确安装的输出：
<pyspark.sql.readwriter.DataFrameReader at 0x10819cd10>
我可以在 scala 中 运行 没有错误，它拉回了我请求的 table：
val gscReadOptionMap = Map(
"url" -> "jdbc:postgresql://12.3.45.678:9101/code",
"user" -> "my_name",
"password" -> "password",
"dbschema" -> "schema",
"dbtable" -> "table",
"partitionColumn" -> "max"
)
val gpdf = spark.read.format("greenplum").options(gscReadOptionMap).load()
它提供以下输出：
gpdf: org.apache.spark.sql.DataFrame = [output_code: string, input_code: string ... 11 more fields]
但是当我尝试将数据库登录到 python 时，我不断收到相同的错误：
import pyspark
print(pyspark.__file__)
from pyspark.sql import DataFrameReader
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext(appName="PythonStreamingQueueStream")
sqlContext = SQLContext(sc)
sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider')
source_df = sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider').options(
url='jdbc:postgresql://12.3.45.678:9101/code',
dbschema='schema',
dbtable = 'table',
user='my_name',
password='password',
driver='org.postgresql.Driver',
partitionColumn='max').load()
sc.stop()
这是错误，通常 python 错误超长：
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/pyspark/sql/readwriter.py", line 165, in load
return self._df(self._jreader.load())
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o111.load.
: java.lang.ClassNotFoundException: Failed to find data source: io.pivotal.greenplum.spark.GreenplumRelationProvider. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:301)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.pivotal.greenplum.spark.GreenplumRelationProvider.DefaultSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:533)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:533)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:533)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:533)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:533)
... 16 more
我有 'greenplum-spark_2.11-1.4.0.jar' 和 'postgresql-42.2.4.jre7.jar' 被带进来...我不确定我做错了什么？
编辑：
我 运行 以 python 两种方式处理罐子，这与错误无关。
import os
os.getcwd()
os.environ['PYSPARK_SUBMIT_ARGS'] = '--master local[*] pyspark-shell --jars /Users/greenplum-spark_2.11-1.4.0.jar, /Users/postgresql-42.2.4.jre7.jar'
我也 运行:
%%bash
export GSC_JAR=/Users/greenplum-spark_2.11-1.4.0.jar export POSTGRES_JAR=/Users/postgresql-42.2.4.jre7.jar
spark-shell --jars ${GSC_JAR}, ${POSTGRES_JAR}
您的 pyspark 似乎没有加载 greenplum-spark_2.11-1.4.0.jar
建议：
检查您的 pyspark 命令是否正在传递带有 greenplum-sparkXXXX.jar 路径的参数“--jars”。下面的示例使用 "GSC_JAR" 环境变量，该变量指向 greenplum-spark_2.11-*.jar 的文件路径和目录“/code”。因此，请对 GSC_JAR 环境变量进行适当的更改。
root@master:/usr/spark-2.1.0#GSC_JAR=$(ls /code/greenplum-spark_2.11-*.jar)
root@master:/usr/spark-2.1.0#pyspark --jars "${GSC_JAR}"
谢谢，
岗
将 Pyspark 与 Greenplum 结合使用的示例：
https://greenplum-spark-connector.readthedocs.io/en/latest/using-pyspark.html
也许这会有所帮助。我使用 pyspark 尝试了 greenplum-spark 连接器文档中现有的 scala 示例。这与您的示例略有不同（我没有使用 SQLContext），但在某一时刻以下对我有用：
user@spark-client$ pyspark --jars
gscPythonOptions = {
"url": "jdbc:postgresql://gpmaster.domain/tutorial",
"user": "user2",
"password": "pivotal",
"dbschema": "faa",
"dbtable": "otp_c",
"partitionColumn": "airlineid"
}
gpdf = spark.read.format("greenplum").options(**gscPythonOptions).load()
之后这些命令对我有用：
gpdf.printSchema()
gpdf.count()
请验证 pyspark 是否加载了 greenplum-spark 连接器 jar
行动：
使用 sc.getConf().getAll() 函数验证 spark.repl.local.jars 是否引用了适当的 jar。
>>> sc.getConf().getAll()
[('spark.app.id', 'app-20180718183929-0000'), ('spark.jars',
'file:///code/usecase1/data/greenplum-spark_2.11-1.4.0.jar'),
('spark.master', 'spark://master:7077'), ('spark.rdd.compress',
'True'), ('spark.driver.host', 'master'),
('spark.serializer.objectStreamReset', '100'),
('spark.repl.local.jars', 'file:///code/usecase1/data/greenplum-
spark_2.11-1.4.0.jar'), ('spark.driver.port', '38611'),
('spark.executor.id', 'driver'), ('spark.submit.deployMode',
'client'), ('spark.app.name', 'PySparkShell'),
('spark.ui.showConsoleProgress', 'true')]
感谢大家的帮助和建议。 pyspark connector / loader 在终端以外的任何界面都不起作用。即 Jupyter 笔记本。
我正在尝试登录服务器并提取数据以在 python 中本地构建模型。我正在使用 pyspark 库 - 但我不断收到相同的错误。这是我 运行ning 的代码，没有错误，所以我知道我已经正确安装了 .jar 文件：
import pyspark
print(pyspark.__file__)
from pyspark.sql import DataFrameReader
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext(appName="PythonStreamingQueueStream")
sqlContext = SQLContext(sc)
sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider')
sc.stop()
它提供此作为指示 jar 已正确安装的输出：
<pyspark.sql.readwriter.DataFrameReader at 0x10819cd10>
我可以在 scala 中 运行 没有错误，它拉回了我请求的 table：
val gscReadOptionMap = Map(
"url" -> "jdbc:postgresql://12.3.45.678:9101/code",
"user" -> "my_name",
"password" -> "password",
"dbschema" -> "schema",
"dbtable" -> "table",
"partitionColumn" -> "max"
)
val gpdf = spark.read.format("greenplum").options(gscReadOptionMap).load()
它提供以下输出：
gpdf: org.apache.spark.sql.DataFrame = [output_code: string, input_code: string ... 11 more fields]
但是当我尝试将数据库登录到 python 时，我不断收到相同的错误：
import pyspark
print(pyspark.__file__)
from pyspark.sql import DataFrameReader
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext(appName="PythonStreamingQueueStream")
sqlContext = SQLContext(sc)
sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider')
source_df = sqlContext.read.format('io.pivotal.greenplum.spark.GreenplumRelationProvider').options(
url='jdbc:postgresql://12.3.45.678:9101/code',
dbschema='schema',
dbtable = 'table',
user='my_name',
password='password',
driver='org.postgresql.Driver',
partitionColumn='max').load()
sc.stop()
这是错误，通常 python 错误超长：
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/pyspark/sql/readwriter.py", line 165, in load
return self._df(self._jreader.load())
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/local/Cellar/apache-spark/2.2.0/libexec/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o111.load.
: java.lang.ClassNotFoundException: Failed to find data source: io.pivotal.greenplum.spark.GreenplumRelationProvider. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:301)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.pivotal.greenplum.spark.GreenplumRelationProvider.DefaultSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:533)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$$anonfun$apply.apply(DataSource.scala:533)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:533)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun.apply(DataSource.scala:533)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:533)
... 16 more
我有 'greenplum-spark_2.11-1.4.0.jar' 和 'postgresql-42.2.4.jre7.jar' 被带进来...我不确定我做错了什么？
编辑： 我 运行 以 python 两种方式处理罐子，这与错误无关。
import os
os.getcwd()
os.environ['PYSPARK_SUBMIT_ARGS'] = '--master local[*] pyspark-shell --jars /Users/greenplum-spark_2.11-1.4.0.jar, /Users/postgresql-42.2.4.jre7.jar'
我也 运行:
%%bash
export GSC_JAR=/Users/greenplum-spark_2.11-1.4.0.jar export POSTGRES_JAR=/Users/postgresql-42.2.4.jre7.jar
spark-shell --jars ${GSC_JAR}, ${POSTGRES_JAR}
您的 pyspark 似乎没有加载 greenplum-spark_2.11-1.4.0.jar
建议： 检查您的 pyspark 命令是否正在传递带有 greenplum-sparkXXXX.jar 路径的参数“--jars”。下面的示例使用 "GSC_JAR" 环境变量，该变量指向 greenplum-spark_2.11-*.jar 的文件路径和目录“/code”。因此，请对 GSC_JAR 环境变量进行适当的更改。
root@master:/usr/spark-2.1.0#GSC_JAR=$(ls /code/greenplum-spark_2.11-*.jar)
root@master:/usr/spark-2.1.0#pyspark --jars "${GSC_JAR}"
谢谢， 岗 将 Pyspark 与 Greenplum 结合使用的示例： https://greenplum-spark-connector.readthedocs.io/en/latest/using-pyspark.html
也许这会有所帮助。我使用 pyspark 尝试了 greenplum-spark 连接器文档中现有的 scala 示例。这与您的示例略有不同（我没有使用 SQLContext），但在某一时刻以下对我有用：
user@spark-client$ pyspark --jars
gscPythonOptions = { "url": "jdbc:postgresql://gpmaster.domain/tutorial", "user": "user2", "password": "pivotal", "dbschema": "faa", "dbtable": "otp_c", "partitionColumn": "airlineid" }
gpdf = spark.read.format("greenplum").options(**gscPythonOptions).load()
之后这些命令对我有用：
gpdf.printSchema()
gpdf.count()
请验证 pyspark 是否加载了 greenplum-spark 连接器 jar
行动： 使用 sc.getConf().getAll() 函数验证 spark.repl.local.jars 是否引用了适当的 jar。
>>> sc.getConf().getAll()
[('spark.app.id', 'app-20180718183929-0000'), ('spark.jars',
'file:///code/usecase1/data/greenplum-spark_2.11-1.4.0.jar'),
('spark.master', 'spark://master:7077'), ('spark.rdd.compress',
'True'), ('spark.driver.host', 'master'),
('spark.serializer.objectStreamReset', '100'),
('spark.repl.local.jars', 'file:///code/usecase1/data/greenplum-
spark_2.11-1.4.0.jar'), ('spark.driver.port', '38611'),
('spark.executor.id', 'driver'), ('spark.submit.deployMode',
'client'), ('spark.app.name', 'PySparkShell'),
('spark.ui.showConsoleProgress', 'true')]
感谢大家的帮助和建议。 pyspark connector / loader 在终端以外的任何界面都不起作用。即 Jupyter 笔记本。