使用 debezium link postgresql 11 无法获取数据库测试的编码

Question

我使用 debezium cdc 连接 pg，我构建了 docker 使用的 pg 11，pg 很好运行。当我在kafka连接器中使用debezium时，它报告：

Couldn't obtain encoding for database test

旋度是：

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST http://localhost:8083/connectors/ -d '{
    "name": "debezium",
    "config": {
        "name": "debezium",
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "localhost",
        "database.port": "5432",
        "database.dbname": "test",
        "database.user": "pg",
        "database.password": "135790",
        "database.server.name": "ls",
        "table.whitelist": "public.test",
        "plugin.name": "pgoutput"
    }
}'

kafka 异常是：

[2020-07-08 09:24:35,076] ERROR Uncaught exception in REST call to /connectors/ (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
java.lang.RuntimeException: Couldn't obtain encoding for database test
    at io.debezium.connector.postgresql.connection.PostgresConnection.determineDatabaseCharset(PostgresConnection.java:434)
    at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:77)
    at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:87)
    at io.debezium.connector.postgresql.PostgresConnector.validate(PostgresConnector.java:102)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:277)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.call(DistributedHerder.java:534)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.call(DistributedHerder.java:531)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:267)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:216)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: FATAL: database "test" does not exist
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
    at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2644)
    at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:137)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:255)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
    at org.postgresql.Driver.makeConnection(Driver.java:458)
    at org.postgresql.Driver.connect(Driver.java:260)
    at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory(JdbcConnection.java:190)
    at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:788)
    at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:783)
    at io.debezium.connector.postgresql.connection.PostgresConnection.determineDatabaseCharset(PostgresConnection.java:431)
    ... 13 more
[2020-07-08 09:24:35,128] INFO 127.0.0.1 - - [08/Jul/2020:01:24:34 +0000] "POST /connectors/ HTTP/1.1" 500 73  330 (org.apache.kafka.connect.runtime.rest.RestServer:60)

Answer 1

在我看来，名为 test 的数据库要么不存在，要么对用户 pg 不可见。

Answer 2

您的有效负载中有一些不准确的地方。保留可解析的名称而不是 localhost。其次，放置正确的数据库命名空间。

        "database.hostname": "FQDN",
        "database.server.name": "test_table_name",

可能是 PostgreSQL 主机未验证 pg 用户的身份验证的情况。它需要进入 pg_hba.conf（在 PostgreSQL 服务器上）并从客户端机器建立 trust/auth，即 Kafka-connector。

# host       DATABASE  USER  ADDRESS                       METHOD  [OPTIONS]
# hostssl    DATABASE  USER  ADDRESS                       METHOD  [OPTIONS]
host         test      pg    Kafka.connector.server.ip/32  md5
hostssl      test      pg    Kafka.connector.server.ip/32  md5

然后，重新启动 PostgreSQL 服务器以获得 pg 用户的有效授权，在我的例子中是 pg_ctl reload.

因为，curl 作为 REST API 调用发生，在 PostgreSQL 服务器的防火墙设置中添加 Kafka-rest(8082) 和 Kafka-connect-rest(8083) 端口。

Answer 3

是的，该消息具有不同的误导性。在我的例子中，问题出在 kafka-connect 和数据库服务器之间的关闭端口中。

使用 debezium link postgresql 11 无法获取数据库测试的编码

Use debezium link postgresql 11 Couldn't obtain encoding for database test

postgresql

apache-kafka

apache-kafka-connect

debezium