Databricks on Apache Spark AttributeError: 'str' object has no attribute '_jvm'

Question

尝试使用 Apache Spark 上的 Databricks 从 Azure Event Hub 读取流数据时出现错误

AttributeError: 'str' object has no attribute '_jvm'

错误详情如下：

----> 8 ehConf['eventhubs.connectionString'] = sparkContext._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)

代码如下：

sparkContext = ""
connectionString = 'Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=both4;SharedAccessKey=adfdMyKeyIGBKYBs=;EntityPath=hubv5'
# Source with default settings
connectionString = connectionString

ehConf = {}

ehConf['eventhubs.connectionString'] = sparkContext._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)

streaming_df = spark \
  .readStream \
  .format("eventhubs") \
  .options(**ehConf) \
  .load()

有没有人遇到过这个错误并找到了解决方案？

Answer 1

不应该是 sparkContext，而应该是 sc:

ehConf['eventhubs.connectionString'] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)

P.S。但是将 built-in Kafka 连接器与 EventHubs 一起使用更容易 - 您不需要安装任何东西，而且性能更高...

Databricks on Apache Spark AttributeError: 'str' object has no attribute '_jvm'

Databricks on Apache Spark AttributeError: 'str' object has no attribute '_jvm'

apache-spark

azure-eventhub

azure-databricks