使用 AWS 数据迁移服务 (DMS) 从 Heroku PostgreSQL 迁移到 AWS RDS 不工作
Migration from Heroku PostgreSQL to AWS RDS using AWS Data Migration Service (DMS) not working
我正在尝试使用 AWS DMS 将数据从 Heroku PostgreSQL 数据库复制和复制到 AWS RDS PostgreSQL 实例，但它不起作用（更多信息见下文）。
在 DMS 日志中我可以看到以下消息：
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
Heroku PostgreSQL 的用户不是超级用户或具有复制角色，但我在 2 天前对同一源数据库进行了实验并且它工作正常（我停止了实验以创建更大的 RDS 实例目标）。我想这次我忘记了一些配置。
- 来源：Heroku PostgreSQL standard-2 (9.5.8)
- 目标：位于 sa-east-1 区域的 AWS RDS PostgreSQL (9.6.3) rds.logical_replication=1
- DMS：us-east-1 区域的实例
完整的 Cloudwatch 日志：
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' running full load and CDC in resume mode after recoverable error, retry #3 (replicationtask.c:1078)
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task Id: bb955197-648a-4177-8653-677b5daf5a6e (replicationtask.c:2682)
2017-10-25T17:43:30 [TASK_MANAGER ]I: Creating threads for all components (replicationtask.c:1664)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Threads for all components were created (replicationtask.c:1832)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Task initialization completed successfully (replicationtask.c:2741)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Going to connect to ODBC connection string: PROTOCOL=7.4-0;DRIVER={PostgreSQL};SERVER=XXXXX.rds.amazonaws.com;DATABASE=XXXXX;PORT=5432;sslmode=verify-ca;pqopt=sslrootcert=/rdsdbdata/rds-metadata/certificates/QQSFSDJTBLFXQLQ6VRVDDAYKCM/dms-ca-cert.pem;UID=USERID; (odbc_endpoint_imp.c:2236)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Target endpoint 'PostgreSQL' is using provider syntax 'PostgreSQLLinux' (provider_syntax_manager.c:589)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Transaction size is 10000. Array size is 1000. (odbc_endpoint_imp.c:837)
2017-10-25T17:43:31 [TARGET_APPLY ]I: CSV files will not be kept (odbc_endpoint_imp.c:1519)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Error CSV files will not be kept (odbc_endpoint_imp.c:1528)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Working in transactional apply mode (endpointshell.c:1436)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Going to connect to ODBC connection string: Driver={PostgreSQL Unicode(x64)};Server=SERVER_AT_HEROKU.compute-1.amazonaws.com;Port=5432;Database=XXXX;UseDeclareFetch=1;Fetch=10000;Uid=USERID;Pwd=PASSWORD;sslmode=require (postgres_endpoint_util.c:252)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Source endpoint 'PostgreSQL Source' is using provider syntax '<default>' (provider_syntax_manager.c:583)
2017-10-25T17:43:34 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:2547)
2017-10-25T17:43:34 [SOURCE_CAPTURE ]I: Initial position is set to 'now' (postgres_endpoint_util.c:343)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' does NOT exist. (postgres_endpoint_capture.c:293)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Initial positioning requested is 'now' (postgres_endpoint_capture.c:313)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Failed to execute statement: 'SELECT xlog_position FROM pg_create_logical_replication_slot('wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e', 'test_decoding')' [1022502] (ar_odbc_stmt.c:2415)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Creating replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' failed. [1022502] (postgres_endpoint_capture.c:334)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Error executing source loop [1022502] (streamcomponent.c:1581)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Task error notification received from subtask 0, thread 0 [1022502] (replicationtask.c:2375)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Stream component failed at subtask 0, component st_0_QVUF6NET6KSB26SCIIRH4UQZAQ [1022502] (subtask.c:1350)
2017-10-25T17:43:35 [TASK_MANAGER ]W: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' encountered a recoverable error (repository.c:4749)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Stream component 'st_0_QVUF6NET6KSB26SCIIRH4UQZAQ' terminated [1022502] (subtask.c:1513)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Subtask #0 ended (replicationtask_util.c:937)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Task management thread terminated (replicationtask.c:3210)
The prereqs for DMS 声明源数据库的用户需要具有超级用户属性； Heroku 不允许这样做。
也许您可以在安装了 heroku cli 的 docker 容器中创建一个 lambda，您可以添加：
heroku pg:backups:capture
heroku pg:backups:download
到 lambda 脚本。然后您可以将 zip 上传到 s3，然后启动脚本将数据加载到 RDS
我正在尝试使用 AWS DMS 将数据从 Heroku PostgreSQL 数据库复制和复制到 AWS RDS PostgreSQL 实例，但它不起作用（更多信息见下文）。
在 DMS 日志中我可以看到以下消息：
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
Heroku PostgreSQL 的用户不是超级用户或具有复制角色，但我在 2 天前对同一源数据库进行了实验并且它工作正常（我停止了实验以创建更大的 RDS 实例目标）。我想这次我忘记了一些配置。
- 来源：Heroku PostgreSQL standard-2 (9.5.8)
- 目标：位于 sa-east-1 区域的 AWS RDS PostgreSQL (9.6.3) rds.logical_replication=1
- DMS：us-east-1 区域的实例
完整的 Cloudwatch 日志：
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' running full load and CDC in resume mode after recoverable error, retry #3 (replicationtask.c:1078)
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task Id: bb955197-648a-4177-8653-677b5daf5a6e (replicationtask.c:2682)
2017-10-25T17:43:30 [TASK_MANAGER ]I: Creating threads for all components (replicationtask.c:1664)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Threads for all components were created (replicationtask.c:1832)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Task initialization completed successfully (replicationtask.c:2741)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Going to connect to ODBC connection string: PROTOCOL=7.4-0;DRIVER={PostgreSQL};SERVER=XXXXX.rds.amazonaws.com;DATABASE=XXXXX;PORT=5432;sslmode=verify-ca;pqopt=sslrootcert=/rdsdbdata/rds-metadata/certificates/QQSFSDJTBLFXQLQ6VRVDDAYKCM/dms-ca-cert.pem;UID=USERID; (odbc_endpoint_imp.c:2236)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Target endpoint 'PostgreSQL' is using provider syntax 'PostgreSQLLinux' (provider_syntax_manager.c:589)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Transaction size is 10000. Array size is 1000. (odbc_endpoint_imp.c:837)
2017-10-25T17:43:31 [TARGET_APPLY ]I: CSV files will not be kept (odbc_endpoint_imp.c:1519)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Error CSV files will not be kept (odbc_endpoint_imp.c:1528)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Working in transactional apply mode (endpointshell.c:1436)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Going to connect to ODBC connection string: Driver={PostgreSQL Unicode(x64)};Server=SERVER_AT_HEROKU.compute-1.amazonaws.com;Port=5432;Database=XXXX;UseDeclareFetch=1;Fetch=10000;Uid=USERID;Pwd=PASSWORD;sslmode=require (postgres_endpoint_util.c:252)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Source endpoint 'PostgreSQL Source' is using provider syntax '<default>' (provider_syntax_manager.c:583)
2017-10-25T17:43:34 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:2547)
2017-10-25T17:43:34 [SOURCE_CAPTURE ]I: Initial position is set to 'now' (postgres_endpoint_util.c:343)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' does NOT exist. (postgres_endpoint_capture.c:293)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Initial positioning requested is 'now' (postgres_endpoint_capture.c:313)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Failed to execute statement: 'SELECT xlog_position FROM pg_create_logical_replication_slot('wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e', 'test_decoding')' [1022502] (ar_odbc_stmt.c:2415)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Creating replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' failed. [1022502] (postgres_endpoint_capture.c:334)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Error executing source loop [1022502] (streamcomponent.c:1581)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Task error notification received from subtask 0, thread 0 [1022502] (replicationtask.c:2375)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Stream component failed at subtask 0, component st_0_QVUF6NET6KSB26SCIIRH4UQZAQ [1022502] (subtask.c:1350)
2017-10-25T17:43:35 [TASK_MANAGER ]W: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' encountered a recoverable error (repository.c:4749)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Stream component 'st_0_QVUF6NET6KSB26SCIIRH4UQZAQ' terminated [1022502] (subtask.c:1513)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Subtask #0 ended (replicationtask_util.c:937)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Task management thread terminated (replicationtask.c:3210)
The prereqs for DMS 声明源数据库的用户需要具有超级用户属性； Heroku 不允许这样做。
也许您可以在安装了 heroku cli 的 docker 容器中创建一个 lambda，您可以添加：
heroku pg:backups:capture
heroku pg:backups:download
到 lambda 脚本。然后您可以将 zip 上传到 s3，然后启动脚本将数据加载到 RDS