将 Debezium Connector for Oracle 与 LogMiner 一起使用，但不使用 XSTREAM

Question

我正在尝试使用 LogMiner 适配器为 oracle 创建一个连接器。我以这种方式预先配置了我的 oracle 数据库。我的 dockerfile

FROM store/oracle/database-enterprise:12.2.0.1

# creating directory inside the container, where all sql scripts will be locaited
#WORKDIR /home/oracle/setup/custom_scripts

WORKDIR /opt/oracle/oradata

WORKDIR /opt/oracle/oradata/recovery_area

WORKDIR /opt/oracle/oradata/ORCLCDB
WORKDIR /opt/oracle/oradata/ORCLCDB/ORCLPDB1/

WORKDIR /u01/app/oracle/product/12.2.0/dbhome_1/inventory
WORKDIR /u01/app/oracle/product/12.2.0/dbhome_1/bin

#Running my custom scripts
COPY configDBora.sh /home/oracle/setup/

RUN chgrp 54321 /opt/oracle/oradata
RUN chown 54321 /opt/oracle/oradata

RUN chgrp 54321 /opt/oracle/oradata/recovery_area
RUN chown 54321 /opt/oracle/oradata/recovery_area

RUN chgrp 54321 /u01/app/oracle/product/12.2.0/dbhome_1/inventory
RUN chown 54321 /u01/app/oracle/product/12.2.0/dbhome_1/inventory

RUN chgrp 54321 /u01/app/oracle/product/12.2.0/dbhome_1/bin
RUN chown 54321 /u01/app/oracle/product/12.2.0/dbhome_1/bin

我的configDBora.sh基于Debezium oracle set up，我在构建它时复制到图像

echo "STAGE 1"
   sqlplus /nolog 2>&1 <<EOF
    CONNECT sys/Admin123 AS SYSDBA
    alter system set db_recovery_file_dest_size = 15G;
    alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile;
    shutdown immediate
    startup mount
    alter database archivelog;
    alter database open;
    archive log list
    exit;
EOF
echo "STAGE 2"
   # Enable LogMiner required database features/settings
   sqlplus sys/Admin123 as sysdba <<- 'EOF'
     ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
     ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

     SELECT SUPPLEMENTAL_LOG_DATA_MIN min,
            SUPPLEMENTAL_LOG_DATA_PK pk,
            SUPPLEMENTAL_LOG_DATA_UI ui,
            SUPPLEMENTAL_LOG_DATA_FK fk,
            SUPPLEMENTAL_LOG_DATA_ALL "all"
         from v$database;
     ALTER PROFILE DEFAULT LIMIT FAILED_LOGIN_ATTEMPTS UNLIMITED;
     exit;
EOF
echo "STAGE 3"
   # Create Log Miner Tablespace and User
   sqlplus sys/Admin123 as sysdba <<- EOF
     CREATE TABLESPACE LOGMINER_TBS DATAFILE '/opt/oracle/oradata/ORCLCDB/logminer_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
     exit;
EOF

   sqlplus sys/Admin123 as sysdba <<- EOF
     alter session set container=ORCLPDB1;
     CREATE TABLESPACE LOGMINER_TBS DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCLPDB1/logminer_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
     exit;
EOF
echo "STAGE 4"
   sqlplus sys/Admin123 as sysdba <<- 'EOF' #THE  MOST IMPORTANT PART
     CREATE USER c##dbzuser IDENTIFIED BY dbz DEFAULT TABLESPACE LOGMINER_TBS QUOTA UNLIMITED ON LOGMINER_TBS CONTAINER=ALL;
     GRANT CREATE SESSION TO c##dbzuser CONTAINER=ALL;
     GRANT SET CONTAINER TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$DATABASE TO c##dbzuser CONTAINER=ALL;
     GRANT FLASHBACK ANY TABLE TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ANY TABLE TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
     GRANT EXECUTE_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ANY TRANSACTION TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ANY DICTIONARY TO c##dbzuser CONTAINER=ALL;
     GRANT LOGMINING TO c##dbzuser CONTAINER=ALL;
     GRANT CREATE TABLE TO c##dbzuser CONTAINER=ALL;
     GRANT LOCK ANY TABLE TO c##dbzuser CONTAINER=ALL;
     GRANT CREATE SEQUENCE TO c##dbzuser CONTAINER=ALL;
     GRANT EXECUTE ON DBMS_LOGMNR TO c##dbzuser CONTAINER=ALL;
     GRANT EXECUTE ON DBMS_LOGMNR_D TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$LOGMNR_LOGS TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$LOGMNR_CONTENTS TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$LOGFILE TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$ARCHIVED_LOG TO c##dbzuser CONTAINER=ALL;
     GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO c##dbzuser CONTAINER=ALL;
     exit;
EOF
echo "STAGE 5"
   sqlplus sys/Admin123 as sysdba <<- EOF
     alter session set container=ORCLPDB1;
     CREATE USER debezium IDENTIFIED BY dbz;
     GRANT CONNECT TO debezium;
     GRANT CREATE SESSION TO debezium;
     GRANT CREATE TABLE TO debezium;
     GRANT CREATE SEQUENCE to debezium;
     ALTER USER debezium QUOTA 100M on users;
     exit;
EOF
echo "STAGE 6"
   sqlplus sys/Admin123 as sysdba <<- EOF
     alter session set current_schema = C##DBZUSER;
     exit;
EOF

我的源 CDC 连接器：

    "tasks.max": 1,
    "connector.class": "io.debezium.connector.oracle.OracleConnector",
    "database.server.name": "server1",
    "database.hostname" : "oracle",
    "database.port" : "1521",
    "database.user" : "c##dbzuser",
    "database.password" : "dbz",
    "database.dbname" : "ORCLCDB",
    "database.history.kafka.bootstrap.servers" : "broker:29092",
    "database.history.kafka.topic": "server1.oracle.history",
    "database.history.skip.unparseable.ddl": "true",
    "include.schema.changes": "true",
    "table.include.list": "C##DBZUSER.*",
    "database.connection.adapter": "logminer",
    "database.tablename.case.insensitive": "true",
    "database.url": "jdbc:oracle:thin:@oracle:1521:ORCLCDB",
    "snapshot.mode": "initial",
    "errors.log.enable": "true",
    
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://schema-registry:8081",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "http://schema-registry:8081"

连接器成功启动并使用我的 table 生成主题。

当我尝试在 oracle 数据库上插入任何 table 时，没有捕获插入（或更新）查询。谁能告诉我哪里出错了？我真的需要帮助解决这个问题。

Answer 1

我决定将我的数据库迁移到 Oracle 19c 企业版的第一步。我从 repo based on this debezium repo 添加了 setup-logminer.sh。当我构建新的 docker 图像时，我将 setup-logminer.sh 复制到此路径 /opt/oracle/scripts/extensions/startup/。

FROM container-registry.oracle.com/database/enterprise:19.3.0.0

WORKDIR /opt/oracle/oradata

WORKDIR /opt/oracle/oradata/recovery_area

WORKDIR /opt/oracle/oradata/ORCLCDB
WORKDIR /opt/oracle/oradata/ORCLCDB/ORCLPDB1/

WORKDIR /u01/app/oracle/product/19c/dbhome_1/inventory
WORKDIR /u01/app/oracle/product/19c/dbhome_1/bin

#Running my custom scripts
COPY setup-logminer.sh /opt/oracle/scripts/extensions/startup/

RUN chgrp 54321 /opt/oracle/oradata
RUN chown 54321 /opt/oracle/oradata

RUN chgrp 54321 /opt/oracle/oradata/recovery_area
RUN chown 54321 /opt/oracle/oradata/recovery_area

RUN chgrp 54321 /u01/app/oracle/product/19c/dbhome_1/inventory
RUN chown 54321 /u01/app/oracle/product/19c/dbhome_1/inventory

RUN chgrp 54321 /u01/app/oracle/product/19c/dbhome_1/bin
RUN chown 54321 /u01/app/oracle/product/19c/dbhome_1/bin

我修改了一点 setup-logminer.sh 的原始脚本，因为我在连接器日志中有这个错误 (STRUCT) type doesn't have a mapping to the SQL database column type。

之后我不得不 SET AUTOCOMMIT ON。默认情况下，Oracle AUTOCOMMIT 是关闭的，我不得不打开它。当我在 IDE 中工作时，我必须始终设置 AUTO（看图片），因为它始终是手动的。 P.S。我还在 Oracle 12c 中打开了自动提交，但没有结果，我决定完全放弃它并迁移到 Oracle 19c。

3。我将我的数据库从 CDB 移到了 PDB，因为在 CDB 中工作很糟糕。您必须使用可插拔数据库。

{
...
    "table.include.list": "DEBEZIUM.*",
...
}

将 Debezium Connector for Oracle 与 LogMiner 一起使用，但不使用 XSTREAM

Using Debezium Connector for Oracle with LogMiner and without XSTREAM

oracle

apache-kafka

apache-kafka-connect

debezium