配置 debezium 连接器以读取 mysql 个没有任何部分数据的二进制日志

Question

我是运行debezium to capture data from mysql bin logs and publish on Kafka using kafka-connect。我今天收到这个错误：

[2019-06-18 11:32:38,175] ERROR Failed to properly convert data value for 'wallet_db_v3.user_registrations.msisdn' of type VARCHAR for row [824, 2, 2019-06-18T06:20:26]: (io.debezium.relational.TableSchemaBuilder:253)
java.lang.IllegalArgumentException: Unexpected value for JDBC type 12 and column msisdn VARCHAR(50) CHARSET utf8 NOT NULL: class=class java.lang.Integer
at io.debezium.jdbc.JdbcValueConverters.handleUnknownData(JdbcValueConverters.java:1164)
at io.debezium.jdbc.JdbcValueConverters.convertValue(JdbcValueConverters.java:1207)
at io.debezium.connector.mysql.MySqlValueConverters.convertString(MySqlValueConverters.java:380)
at io.debezium.connector.mysql.MySqlValueConverters.lambda$converter(MySqlValueConverters.java:293)
at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator(TableSchemaBuilder.java:249)
at io.debezium.relational.TableSchema.valueFromColumnData(TableSchema.java:145)
at io.debezium.connector.mysql.RecordMakers.insert(RecordMakers.java:244)
at io.debezium.connector.mysql.RecordMakers$RecordsForTable.create(RecordMakers.java:456)
at io.debezium.connector.mysql.BinlogReader.handleInsert(BinlogReader.java:833)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)

table结构：

'id', 'int(10) unsigned', 'NO', 'PRI', NULL, 'auto_increment'
'msisdn', 'varchar(50)', 'NO', '', NULL, ''
'device_id', 'varchar(50)', 'NO', '', NULL, ''
'status', 'enum(\'waiting_for_validation\',\'approved\',\'rejected\',\'expired\')', 'NO', '', NULL, ''
'rejection_reason', 'enum(\'max_otp_validations_reached\',\'canceled_by_user\')', 'YES', '', NULL, ''
'updated_at', 'datetime', 'YES', '', 'CURRENT_TIMESTAMP', 'on update CURRENT_TIMESTAMP'
'created_at', 'datetime', 'NO', '', NULL, ''
'channel', 'varchar(50)', 'YES', '', NULL, ''

整行应如下所示：

select * from user_registrations where id = 824;
['824', '+115004937263', '56DA9A2FD4D91320D01514ED497C719E5DC9323F', 'approved', NULL, '2019-06-18 06:20:26', '2019-06-18 06:20:23', NULL]

如果现在查询 table，则此行显示为完整。

我怀疑 debezium 连接器在记录完全写入 mysql binlog 之前正在读取记录。有没有人配置 debezium 以确保在记录完全写入 binlog

后读取

my.cnf设置参考：

log-bin=/var/log/mysql/mysql-bin
binlog_format=row
binlog_row_image=full

Answer 1

问题出在来源 table - NDB table 不受 binlog_row_image 参数设置为 [=11] 的控制=].

配置 debezium 连接器以读取 mysql 个没有任何部分数据的二进制日志

Configure debezium connector to read mysql binary logs without any partial data

mysql

apache-kafka-connect

debezium