如何使用 confluent-kafka-python 从 Data Science Experience / Spark 即服务连接到 Message Hub?
How to connect to Message Hub from Data Science Experience / Spark as a Service using confluent-kafka-python?
Bluemix MessageHub 文档将 python 用户指向融合的 kafka 库:
所以我尝试安装:
!pip install --user confluent-kafka
但是,我遇到了这个错误:
Collecting confluent-kafka
Using cached confluent-kafka-0.9.1.2.tar.gz
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... - \ error
Complete output from command /usr/local/src/bluemix_jupyter_bundle.v22/notebook/bin/python -u -c "import setuptools, tokenize;__file__='/gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /gpfs/fs01/user/xxxx/notebook/tmp/pip-PyAwq2-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/confluent_kafka
copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka
creating build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/confluent_kafka
creating build/temp.linux-x86_64-2.7/confluent_kafka/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/src/bluemix_jupyter_bundle.v22/notebook/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:0:
confluent_kafka/src/confluent_kafka.h:20:32: fatal error: librdkafka/rdkafka.h: No such file or directory
#include <librdkafka/rdkafka.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/src/bluemix_jupyter_bundle.v22/notebook/bin/python -u -c "import setuptools, tokenize;__file__='/gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /gpfs/fs01/user/xxxx/notebook/tmp/pip-PyAwq2-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/
根据 confluent readme,您需要先安装 librdkafka 本机库,然后才能使用他们的 Python 客户端。
正如您在其他评论中发现的那样,它们是另一种纯 Python 库 kafka-python,您可能会发现它更易于用于 Bluemix 应用程序
Bluemix MessageHub 文档将 python 用户指向融合的 kafka 库:
所以我尝试安装:
!pip install --user confluent-kafka
但是,我遇到了这个错误:
Collecting confluent-kafka
Using cached confluent-kafka-0.9.1.2.tar.gz
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... - \ error
Complete output from command /usr/local/src/bluemix_jupyter_bundle.v22/notebook/bin/python -u -c "import setuptools, tokenize;__file__='/gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /gpfs/fs01/user/xxxx/notebook/tmp/pip-PyAwq2-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/confluent_kafka
copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka
creating build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/confluent_kafka
creating build/temp.linux-x86_64-2.7/confluent_kafka/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/src/bluemix_jupyter_bundle.v22/notebook/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
In file included from confluent_kafka/src/confluent_kafka.c:17:0:
confluent_kafka/src/confluent_kafka.h:20:32: fatal error: librdkafka/rdkafka.h: No such file or directory
#include <librdkafka/rdkafka.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/local/src/bluemix_jupyter_bundle.v22/notebook/bin/python -u -c "import setuptools, tokenize;__file__='/gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /gpfs/fs01/user/xxxx/notebook/tmp/pip-PyAwq2-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /gpfs/global_fs01/sym_shared/YPProdSpark/user/xxxx/notebook/tmp/pip-build-N3zDUh/confluent-kafka/
根据 confluent readme,您需要先安装 librdkafka 本机库,然后才能使用他们的 Python 客户端。
正如您在其他评论中发现的那样,它们是另一种纯 Python 库 kafka-python,您可能会发现它更易于用于 Bluemix 应用程序