Debezium 失败 - 无法复制，因为主清除了所需的二进制日志
Debezium failed - Cannot replicate because the master purged required binary logs
我正在使用 debezium-kafka 来捕获 MySQL 中的变化。 MySQL 我使用的是 AWS RDS。
我有一个启用了 GTID 的只读副本配置。我已将 debezium 与副本连接起来。
但是 运行 几天后，debezium 失败并出现错误：
org.apache.kafka.connect.errors.ConnectException: Cannot replicate because the master purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. To find the missing transactions, see the master's error log or the manual for GTID_SUBTRACT. Error code: 1236; SQLSTATE: HY000.
tat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
tat io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)
tat io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:1018)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.run(BinaryLogClient.java:825)
tat java.base/java.lang.Thread.run(Thread.java:834)
aused by: com.github.shyiko.mysql.binlog.network.ServerException: Cannot replicate because the master purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. To find the missing transactions, see the master's error log or the manual for GTID_SUBTRACT.
tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:914)
t... 3 more
我无法弄清楚是什么导致了问题，MYSQL 主服务器和副本服务器都正常，都已启动并且 运行。那为什么 debezium 失败了？
好的，我找到了解决方案，我正在使用 Amazon RDS MySQL。 Amazon RDS 通常会尽快清除二进制日志。由于 GTID 在被 Debezium 消耗之前被清除。
指定 RDS 保留二进制文件的小时数：
call mysql.rds_set_configuration('binlog retention hours', 24);
Link: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html
我正在使用 debezium-kafka 来捕获 MySQL 中的变化。 MySQL 我使用的是 AWS RDS。 我有一个启用了 GTID 的只读副本配置。我已将 debezium 与副本连接起来。
但是 运行 几天后，debezium 失败并出现错误：
org.apache.kafka.connect.errors.ConnectException: Cannot replicate because the master purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. To find the missing transactions, see the master's error log or the manual for GTID_SUBTRACT. Error code: 1236; SQLSTATE: HY000.
tat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
tat io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)
tat io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:1018)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
tat com.github.shyiko.mysql.binlog.BinaryLogClient.run(BinaryLogClient.java:825)
tat java.base/java.lang.Thread.run(Thread.java:834)
aused by: com.github.shyiko.mysql.binlog.network.ServerException: Cannot replicate because the master purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. To find the missing transactions, see the master's error log or the manual for GTID_SUBTRACT.
tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:914)
t... 3 more
我无法弄清楚是什么导致了问题，MYSQL 主服务器和副本服务器都正常，都已启动并且 运行。那为什么 debezium 失败了？
好的，我找到了解决方案，我正在使用 Amazon RDS MySQL。 Amazon RDS 通常会尽快清除二进制日志。由于 GTID 在被 Debezium 消耗之前被清除。
指定 RDS 保留二进制文件的小时数：
call mysql.rds_set_configuration('binlog retention hours', 24);
Link: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html