使用 Debezium 通过 SSL 连接到云 SQL 时出错
Error connecting to Cloud SQL with SSL using Debezium
Objective:使用 debezium 从云端捕获变化 SQL。 Cloud SQL 的实例已根据说明启用 SSL here
场景:我在本地机器上有 debezium connect、kafka 和 zookpeer 运行 作为 docker 容器。我已经针对没有 SSL 的 Cloud SQL 实例测试了设置。一切正常。启用 SSL 后,将 pem 文件(server-ca.pem
、client-cert.pem
、client-key.pem
)转换为密钥库和信任库,将它们作为文件安装在 debezium connect docker 容器中,我得到Debezium 容器日志中的错误(在将 POST 请求发送到端点之后):
org.apache.kafka.connect.errors.ConnectException: Error reading MySQL variables: Access denied for user 'user'@'redacted_my_ip' (using password: YES)
at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:297)
at io.debezium.connector.mysql.MySqlJdbcContext.readMySqlSystemVariables(MySqlJdbcContext.java:278)
at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:81)
at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:53)
at io.debezium.connector.mysql.MySqlConnectorTask.createAndStartTaskContext(MySqlConnectorTask.java:331)
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:136)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:47)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:198)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Access denied for user 'user'@'redacted_my_ip' (using password: YES)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:207)
at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory(JdbcConnection.java:179)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:756)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:751)
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:298)
at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:284)
... 14 more
我的初步分析:我一直在查看TRACE日志和源代码。根据日志,可以成功测试连接
2019-04-10 10:11:45,777 INFO || Successfully tested connection for jdbc:mysql://redacted_ip:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=true&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL with user 'user' [io.debezium.connector.mysql.MySqlConnector]
此日志是在 line 101
之后立即生成的
这里我们对数据库执行查询,只有成功执行才会让控制流向日志。根据我的说法,这意味着 debezium connect 能够连接到数据库,但在其他地方失败了。根据堆栈跟踪,debezium 第二次连接失败,here
我发送的payload如下:
{
"name": "connector-test",
"config": {
"connector.class": "MySql",
"tasks.max": "1",
"database.hostname": "redacted_ip",
"database.port": "3306",
"database.user": "user",
"database.password": "redacted_user_password",
"database.server.name": "dbserver1",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "dbhistory.inventory",
"database.ssl.mode": "required",
"database.ssl.keystore": "./keystore",
"database.ssl.keystore.password": "redacted_keystore_password",
"database.ssl.truststore": "./truststore",
"database.ssl.truststore.password": "redacted_truststore_password"
}
}
需要哪些步骤才能使上述设置正常工作
此问题已得到修复,现在已在主分支上与 PR
合并
Objective:使用 debezium 从云端捕获变化 SQL。 Cloud SQL 的实例已根据说明启用 SSL here
场景:我在本地机器上有 debezium connect、kafka 和 zookpeer 运行 作为 docker 容器。我已经针对没有 SSL 的 Cloud SQL 实例测试了设置。一切正常。启用 SSL 后,将 pem 文件(server-ca.pem
、client-cert.pem
、client-key.pem
)转换为密钥库和信任库,将它们作为文件安装在 debezium connect docker 容器中,我得到Debezium 容器日志中的错误(在将 POST 请求发送到端点之后):
org.apache.kafka.connect.errors.ConnectException: Error reading MySQL variables: Access denied for user 'user'@'redacted_my_ip' (using password: YES)
at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:297)
at io.debezium.connector.mysql.MySqlJdbcContext.readMySqlSystemVariables(MySqlJdbcContext.java:278)
at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:81)
at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:53)
at io.debezium.connector.mysql.MySqlConnectorTask.createAndStartTaskContext(MySqlConnectorTask.java:331)
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:136)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:47)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:198)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Access denied for user 'user'@'redacted_my_ip' (using password: YES)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:207)
at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory(JdbcConnection.java:179)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:756)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:751)
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:298)
at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:284)
... 14 more
我的初步分析:我一直在查看TRACE日志和源代码。根据日志,可以成功测试连接
2019-04-10 10:11:45,777 INFO || Successfully tested connection for jdbc:mysql://redacted_ip:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=true&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL with user 'user' [io.debezium.connector.mysql.MySqlConnector]
此日志是在 line 101
之后立即生成的这里我们对数据库执行查询,只有成功执行才会让控制流向日志。根据我的说法,这意味着 debezium connect 能够连接到数据库,但在其他地方失败了。根据堆栈跟踪,debezium 第二次连接失败,here
我发送的payload如下:
{
"name": "connector-test",
"config": {
"connector.class": "MySql",
"tasks.max": "1",
"database.hostname": "redacted_ip",
"database.port": "3306",
"database.user": "user",
"database.password": "redacted_user_password",
"database.server.name": "dbserver1",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "dbhistory.inventory",
"database.ssl.mode": "required",
"database.ssl.keystore": "./keystore",
"database.ssl.keystore.password": "redacted_keystore_password",
"database.ssl.truststore": "./truststore",
"database.ssl.truststore.password": "redacted_truststore_password"
}
}
需要哪些步骤才能使上述设置正常工作
此问题已得到修复,现在已在主分支上与 PR
合并