Debezium Kafka 连接器 mongodb：将 kafka 连接器连接到 mongodb 时出错

Question

下面是我在 /etc/kafka/connect-mongodb-source.properties

中的 MongoDB 配置

name=mongodb-source-connector 
connector.class=io.debezium.connector.mongodb.MongoDbConnector 
mongodb.hosts=/remoteserveraddress:27017 
mongodb.name=mongo_conn 
initial.sync.max.threads=1 
tasks.max=1

但低于错误

 ERROR Plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5a058be5 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)

运行连接器处于独立模式。

我在 debezium-debezium-connector-mongodb-1.0.0/debezium-connector-mongodb-1.0.[=38 下有 mongodbconnector class =]

也class路径设置如下

#for CLASSPATH
CLASSPATH=/Users/111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/*
export CLASSPATH
PATH=$PATH:/usr/local/sbin
export PATH

使用插件路径我看到它能够注册和加载所有必需的插件..

[2020-01-10 08:14:07,916] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectStandalone:78)
[2020-01-10 08:14:07,942] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongodb-driver-3.11.1.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,082] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongodb-driver-3.11.1.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-01-10 08:14:08,083] INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,083] INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,083] INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,085] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/debezium-connector-mongodb-1.0.0.Final.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,120] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/debezium-connector-mongodb-1.0.0.Final.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-01-10 08:14:08,121] INFO Added plugin 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,121] INFO Added plugin 'io.debezium.connector.mongodb.transforms.ExtractNewDocumentState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,121] INFO Added plugin 'io.debezium.connector.mongodb.transforms.UnwrapFromMongoDbEnvelope' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,122] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/debezium-core-1.0.0.Final.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,198] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/debezium-core-1.0.0.Final.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-01-10 08:14:08,198] INFO Added plugin 'io.debezium.converters.ByteBufferConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,199] INFO Added plugin 'io.debezium.transforms.UnwrapFromEnvelope' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,199] INFO Added plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,199] INFO Added plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,199] INFO Added plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,200] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongo-kafka-0.2-all.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,340] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongo-kafka-0.2-all.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-01-10 08:14:08,340] INFO Added plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,340] INFO Added plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-01-10 08:14:08,341] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/bson-3.11.1.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,373] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/bson-3.11.1.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-01-10 08:14:08,373] INFO Loading plugin from: /Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongodb-driver-core-3.11.1.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2020-01-10 08:14:08,465] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/1111111/workspace/KafkaConnect/confluent-5.3.2/debezium-debezium-connector-mongodb-1.0.0/mongodb-driver-core-3.11.1.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)

但最后又是同样的问题

    [2020-01-10 08:40:43,613] ERROR Plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@33f2df51 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
[2020-01-10 08:40:43,809] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121)
java.lang.IllegalArgumentException: username can not be null
    at com.mongodb.MongoCredential.<init>(MongoCredential.java:350)
    at com.mongodb.MongoCredential.<init>(MongoCredential.java:344)
    at com.mongodb.MongoCredential.createCredential(MongoCredential.java:169)
    at io.debezium.connector.mongodb.ConnectionContext.<init>(ConnectionContext.java:69)
    at io.debezium.connector.mongodb.MongoDbConnector.validate(MongoDbConnector.java:222)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:313)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)
[2020-01-10 08:40:43,810] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)

>curl localhost:8083/connector-plugins

的输出

[{"class":"com.mongodb.kafka.connect.MongoSinkConnector","type":"sink","version":"0.2"},{"class":"com.mongodb.kafka.connect.MongoSourceConnector","type":"source","version":"0.2"},{"class":"io.confluent.connect.activemq.ActiveMQSourceConnector","type":"source","version":"5.3.2"},{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","type":"sink","version":"5.3.2"},{"class":"io.confluent.connect.ibm.mq.IbmMQSourceConnector","type":"source","version":"5.3.2"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"5.3.2"},{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"5.3.2"},{"class":"io.confluent.connect.jms.JmsSourceConnector","type":"source","version":"5.3.2"},{"class":"io.confluent.connect.replicator.ReplicatorSourceConnector","type":"source","version":"5.3.2"},{"class":"io.confluent.connect.s3.S3SinkConnector","type":"sink","version":"5.3.2"},{"class":"io.confluent.connect.storage.tools.SchemaSourceConnector","type":"source","version":"5.3.2-ce"},{"class":"io.debezium.connector.mongodb.MongoDbConnector","type":"source","version":"1.0.0.Final"},{"class":"org.apache.kafka.connect.file.FileStr

提前致谢

Answer 1

首先，请使用 Kafka Connect REST 接口检查插件的安装情况（查看详情 here）。

尝试使用 plugin path 机制而不是 CLASSPATH 安装 Kafka Connect 插件（更多信息在 docs 中）。

Answer 2

有两个问题

Debezium 连接器硬编码了 source='admin'，无法覆盖它，我尝试连接的数据库服务器没有任何管理员作为身份验证。
我使用的测试服务器是独立服务器并且没有副本，所以我必须创建副本并将此服务器添加到其中。 Debezium 连接器不支持独立服务器。

Debezium Kafka 连接器 mongodb：将 kafka 连接器连接到 mongodb 时出错

Debezium Kafka connector mongodb : Error connecting kafka connector to mongodb

apache-kafka-connect

debezium

mongodb-kafka-connector