kafka 连接和 kafka 主机要求

Question

我正在使用 Couchbase 水槽连接器。 CB 和 kafka 位于不同 AWS 区域的 2 个不同的 EC2 实例中。我正在关注这些文档：

基于这些，我收集到连接必须运行在也安装了 kafka 的主机上。我可以在远程主机上连接运行，以便我从远程 kafka 读取消息并将消息接收到远程 CB 存储桶中吗？有专门针对此的文档吗？

此外，我收到以下错误：

        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
    at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.3.0', encodedVersion=2.3.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}
    at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:180)
    at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:152)
    at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:519)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:289)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)

Answer 1

I gather that connect must run on a host which also has kafka installed.

有 Kafka 库，是的，但实际上没有运行ning 代理，不。

换句话说，您不能将 Kafka Connect 作为单独的服务下载，因此当您获取它时它已经附带了 Kafka 库。

Can my connect run on a remote host, such that I read from a remote kafka and sink messages into a remote CB bucket?

这是推荐的部署方式。类似于您不会运行经纪人上的正常生产者和消费者。 Kafka Connect 应该同样对待。

bootstrap 服务器和 couchbase 连接字符串将是它们各自的远程地址

文档在有点这里

Also, I am getting following error: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector

如果您查看 available connectors are: 的列表，您会发现您的列表没有列出。这意味着您没有使用您的库正确加载连接器属性的 plugin.path。

正如 Couchbase 文档所述：

Modify the plugin.path property to include the parent directory of kafka-connect-couchbase-<version>.jar

您还可以在发布配置之前使用 /connector-plugins 端点

验证您的连接器是否已加载

Answer 2

Kafka Connect 无法找到您的连接器。您可以将 plugin.path 设置为指向您的连接器 jar 文件，或者您可以将连接器 jar 文件复制到 kafka lib 目录中。

kafka 连接和 kafka 主机要求

kafka connect and kafka host requirements

couchbase

apache-kafka

apache-kafka-connect