Spark 2.1 DB2 驱动问题

Question

在 IBM DSX 上，对于 Python-Spark 2.1 内核，我得到以下代码：

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# @hidden_cell
# The following code is used to access your data and contains your credentials.
# You might want to remove those credentials before you share your notebook.

properties = {
    'jdbcurl': 'JDBCURL',
    'user': 'USER',
    'password': 'PASSWORD'
}

data_df_1 = spark.read.jdbc(properties['jdbcurl'], table='USER.TABLE', properties=properties)
data_df_1.head()

这个错误：

Py4JJavaError: An error occurred while calling o82.jdbc.
: java.sql.SQLException: No suitable driver
    at java.sql.DriverManager.getDriver(DriverManager.java:326)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun.apply(JDBCOptions.scala:84)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun.apply(JDBCOptions.scala:84)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:83)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:34)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
    at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:166)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:507)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:785)

我只在第一次执行时遇到这个错误。但是，当我再次执行相同的代码时它起作用了。

Answer 1

这看起来像 https://issues.apache.org/jira/browse/SPARK-19209。该问题中建议的解决方法是提供 "driver" 选项。

在您的示例中，如果您需要 DB2 驱动程序：

properties = {
   'jdbcurl': 'JDBCURL',
   'user': 'USER',
   'password': 'PASSWORD',
   'driver': 'com.ibm.db2.jcc.DB2Driver'
}

Spark 2.1 DB2 驱动问题

Spark 2.1 DB2 driver Issue

apache-spark

pyspark

data-science-experience