Kafka Connect docker 图像 JDBC 找不到连接插件

Question

Kafka Connect 的新手，在获取基本 JDBC 源代码时遇到问题。我正在使用以下 docker-compose.yml 片段来创建图像：

kafka-connect_fronting:
image: confluentinc/cp-kafka-connect
container_name: kafka-connect_fronting
hostname: connect_fronting
depends_on:
  - zookeeper_fronting
  - kafka_fronting
  - schema-registry_fronting
ports:
  - "8083:8083"
volumes:
  - ./jars:/etc/kafka-connect/jars/
environment:
  CONNECT_BOOTSTRAP_SERVERS: 'kafka_fronting:29092'
  CONNECT_REST_ADVERTISED_HOST_NAME: connect
  CONNECT_REST_PORT: 8083
  CONNECT_GROUP_ID: compose-connect-group
  CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
  CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
  CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
  CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
  CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
  CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
  CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
  CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
  CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry_fronting:8081'
  CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
  CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry_fronting:8081'
  CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
  CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
  CONNECT_ZOOKEEPER_CONNECT: 'zookeeper_fronting:32181'
  CONNECT_PLUGIN_PATH: '/etc/kafka-connect/jars'

图像显示正常启动：

curl http://localhost:8083/

{"version":"1.0.0-cp1","commit":"ec61c5e93da662df"}

并确认图像中存在 mysql jar：

root@connect_fronting:~# ls -la /etc/kafka-connect/jars
total 980
drwxr-xr-x 4 root root    128 Mar 26 13:20 .
drwxrwxrwx 1 root root   4096 Mar 26 13:56 ..
-rw-r--r-- 1 root root 989497 May  4  2016 mysql-connector-java-5.1.39-bin.jar

但是当我尝试创建连接器时，我运行遇到了麻烦：

curl -X POST \
  -H "Content-Type: application/json" \
  --data '{ "name": "quickstart-jdbc-source", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": 1, "connection.url": "jdbc:mysql://127.0.0.1:3306/maintenance?user=root&password=superset", "mode": "incrementing", "incrementing.column.name": "id", "timestamp.column.name": "modified", "topic.prefix": "quickstart-jdbc-", "poll.interval.ms": 1000 } }' \
  http://kafka-connect_fronting:8083/connectors

{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='1.0.0-cp1', encodedVersion=1.0.0-cp1, type=source, typeName='source', location='classpath'}"}

当我 ping API 时，我只看到文件 source/sink 插件：

$curl http://localhost:8083/connector-plugins/
[{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"1.0.0-cp1"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"1.0.0-cp1"}

文档说明包含 JBDC。我一定是遗漏了一些基本的东西，感谢您的帮助。

Answer 1

您的 CONNECT_PLUGIN_PATH 除了存储自定义插件的位置外，还需要包括 Confluent 平台附带的 Connect 插件的位置（我假设您的示例中为 /etc/kafka-connect/jars）。这个属性是一个列表，所以它有多个路径。

我的猜测是：

CONNECT_PLUGIN_PATH: '/usr/share/java,/etc/kafka-connect/jars'

会起作用。但是您必须仔细检查您正在使用的特定 Docker 图像确实存储了 Confluent 的 Connect 插件：/usr/share/java。

docker 快速入门中的更多详细信息：https://docs.confluent.io/current/installation/docker/docs/quickstart.html#kafka-connect

Kafka Connect docker 图像 JDBC 找不到连接插件

Kafka Connect docker image JDBC connect plugin not found

jdbc

apache-kafka

apache-kafka-connect