Debezium MS SQL 服务器连接器问题
Debezium MS SQL Server connector issues
这里是 Kafka 新手。
环境设置
OS: Ubuntu 20.04
Kafka 安装在计算机本地的主文件夹中。 Kafka和Zookeeper服务都是运行.
我正在尝试配置 Kafka 连接以连接到我使用以下命令构建的 SQL 服务器容器中托管的数据库
docker run --name SQL19 -p 1433:1433 -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Password258*" -e "MSSQL_AGENT_ENABLED=True" -v ~/SqlDockerVol/userdatabase:/userdatabase -v ~/SqlDockerVol/sqlbackups:/sqlbackups -d mcr.microsoft.com/mssql/server:2019-lat估计
容器运行可以连接成功
我在数据库级别和 table 级别启用了 CDC。
EXEC sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'WorkQueue',
@role_name = NULL,
@supports_net_changes = 1
连接器属性文件
name= DebeziumSQLServerconnector
connector.class= io.debezium.connector.sqlserver.SqlServerConnector
database.hostname=linuxdesktop
database.port= 1433
database.user= sa
database.password= Password258*
database.dbname= mydb
database.server.name= linuxdesktop
table.include.list= dbo.WorkQueue
database.history.kafka.bootstrap.servers= linuxdesktop:9092
database.history.kafka.topic= dbhistory.mydb
连接-standalone.properties plugin.path
plugin.path=/home/vinu/kafka-server/connector
我将所有 .JAR 文件复制到 plugin.path
然后我尝试使用下面的命令启动连接器
~/kafka-server/bin/connect-standalone.sh ~/kafka-server/config/connect-standalone.properties DebeziumSQLServerconnector.properties
但是进程启动失败，出现如下错误信息
2021-03-15 13:36:36,752] ERROR Failed to create job for
DebeziumSQLServerconnector.properties
(org.apache.kafka.connect.cli.ConnectStandalone:110) [2021-03-15
13:36:36,753] ERROR Stopping after connector error
(org.apache.kafka.connect.cli.ConnectStandalone:121)
java.util.concurrent.ExecutionException:
org.apache.kafka.connect.errors.ConnectException: Failed to find any
class that implements Connector and which name matches
io.debezium.connector.sqlserver.SqlServerConnector , available
connectors are: PluginDesc{klass=class
io.debezium.connector.sqlserver.SqlServerConnector,
name='io.debezium.connector.sqlserver.SqlServerConnector',
version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source,
typeName='source',
location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'},
PluginDesc{klass=class
org.apache.kafka.connect.file.FileStreamSinkConnector,
name='org.apache.kafka.connect.file.FileStreamSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.file.FileStreamSourceConnector,
name='org.apache.kafka.connect.file.FileStreamSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorCheckpointConnector,
name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorHeartbeatConnector,
name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorSourceConnector,
name='org.apache.kafka.connect.mirror.MirrorSourceConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockConnector,
name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0',
encodedVersion=2.7.0, type=connector, typeName='connector',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockSinkConnector,
name='org.apache.kafka.connect.tools.MockSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockSourceConnector,
name='org.apache.kafka.connect.tools.MockSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.SchemaSourceConnector,
name='org.apache.kafka.connect.tools.SchemaSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.VerifiableSinkConnector,
name='org.apache.kafka.connect.tools.VerifiableSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.VerifiableSourceConnector,
name='org.apache.kafka.connect.tools.VerifiableSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'} at
org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
at
org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
at
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to
find any class that implements Connector and which name matches
io.debezium.connector.sqlserver.SqlServerConnector , available
connectors are: PluginDesc{klass=class
io.debezium.connector.sqlserver.SqlServerConnector,
name='io.debezium.connector.sqlserver.SqlServerConnector',
version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source,
typeName='source',
location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'},
PluginDesc{klass=class
org.apache.kafka.connect.file.FileStreamSinkConnector,
name='org.apache.kafka.connect.file.FileStreamSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.file.FileStreamSourceConnector,
name='org.apache.kafka.connect.file.FileStreamSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorCheckpointConnector,
name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorHeartbeatConnector,
name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.mirror.MirrorSourceConnector,
name='org.apache.kafka.connect.mirror.MirrorSourceConnector',
version='1', encodedVersion=1, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockConnector,
name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0',
encodedVersion=2.7.0, type=connector, typeName='connector',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockSinkConnector,
name='org.apache.kafka.connect.tools.MockSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.MockSourceConnector,
name='org.apache.kafka.connect.tools.MockSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.SchemaSourceConnector,
name='org.apache.kafka.connect.tools.SchemaSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.VerifiableSinkConnector,
name='org.apache.kafka.connect.tools.VerifiableSinkConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'}, PluginDesc{klass=class
org.apache.kafka.connect.tools.VerifiableSourceConnector,
name='org.apache.kafka.connect.tools.VerifiableSourceConnector',
version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source',
location='classpath'} at
org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:208)
at
org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:180)
at
org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:572)
at
org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:342)
at
org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig(AbstractHerder.java:326)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
可能是什么问题？
注意
所有 Kafka 组件都安装在本地机器上。我的设置中唯一的容器是 SQL 服务器。
您不得将连接器 JAR 放入 /home/vinu/kafka-server/connector，而应放入 /home/vinu/kafka-server/connector/debezium-sqlserver-connector 或任何其他名称。
连接器未放在 plugin.path 中，但每个连接器放在 plugin.path.
中的单独目录中
available connectors 列出了 class，因此您的插件没问题。
您有一个 space 问题。注意日志输出逗号位置
matches io.debezium.connector.sqlserver.SqlServerConnector ,
从整个 属性 文件中删除尾随白色space 并在值
之前删除前导白色space
这里是 Kafka 新手。
环境设置
OS: Ubuntu 20.04
Kafka 安装在计算机本地的主文件夹中。 Kafka和Zookeeper服务都是运行.
我正在尝试配置 Kafka 连接以连接到我使用以下命令构建的 SQL 服务器容器中托管的数据库
docker run --name SQL19 -p 1433:1433 -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Password258*" -e "MSSQL_AGENT_ENABLED=True" -v ~/SqlDockerVol/userdatabase:/userdatabase -v ~/SqlDockerVol/sqlbackups:/sqlbackups -d mcr.microsoft.com/mssql/server:2019-lat估计
容器运行可以连接成功
我在数据库级别和 table 级别启用了 CDC。
EXEC sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'WorkQueue',
@role_name = NULL,
@supports_net_changes = 1
连接器属性文件
name= DebeziumSQLServerconnector
connector.class= io.debezium.connector.sqlserver.SqlServerConnector
database.hostname=linuxdesktop
database.port= 1433
database.user= sa
database.password= Password258*
database.dbname= mydb
database.server.name= linuxdesktop
table.include.list= dbo.WorkQueue
database.history.kafka.bootstrap.servers= linuxdesktop:9092
database.history.kafka.topic= dbhistory.mydb
连接-standalone.properties plugin.path
plugin.path=/home/vinu/kafka-server/connector
我将所有 .JAR 文件复制到 plugin.path
然后我尝试使用下面的命令启动连接器
~/kafka-server/bin/connect-standalone.sh ~/kafka-server/config/connect-standalone.properties DebeziumSQLServerconnector.properties
但是进程启动失败，出现如下错误信息
2021-03-15 13:36:36,752] ERROR Failed to create job for DebeziumSQLServerconnector.properties (org.apache.kafka.connect.cli.ConnectStandalone:110) [2021-03-15 13:36:36,753] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.sqlserver.SqlServerConnector , available connectors are: PluginDesc{klass=class io.debezium.connector.sqlserver.SqlServerConnector, name='io.debezium.connector.sqlserver.SqlServerConnector', version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source, typeName='source', location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0', encodedVersion=2.7.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115) at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118) Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.sqlserver.SqlServerConnector , available connectors are: PluginDesc{klass=class io.debezium.connector.sqlserver.SqlServerConnector, name='io.debezium.connector.sqlserver.SqlServerConnector', version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source, typeName='source', location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0', encodedVersion=2.7.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:208) at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:180) at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:572) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:342) at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig(AbstractHerder.java:326) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
可能是什么问题？
注意 所有 Kafka 组件都安装在本地机器上。我的设置中唯一的容器是 SQL 服务器。
您不得将连接器 JAR 放入 /home/vinu/kafka-server/connector，而应放入 /home/vinu/kafka-server/connector/debezium-sqlserver-connector 或任何其他名称。
连接器未放在 plugin.path 中，但每个连接器放在 plugin.path.
available connectors 列出了 class，因此您的插件没问题。
您有一个 space 问题。注意日志输出逗号位置
matches io.debezium.connector.sqlserver.SqlServerConnector ,
从整个 属性 文件中删除尾随白色space 并在值
之前删除前导白色space