19c 上的外部 table 读取问题

External table read issue on 19c

我们正在从 Oracle 11g -> 19 迁移数据库并面临外部问题 table。旧数据库和新数据库具有完全相同的 table 定义并指向同一个文件(不同主机上的数据库 运行 但指向相同的 qtree)。旧数据库可以毫无错误地查询文件,但新数据库会拒绝所有行: KUP-04023:字段开始在记录结束之后

表格配置如下:

CREATE TABLE TEST
(
    AA    VARCHAR2 (40 BYTE),
    BB    VARCHAR2 (2 BYTE),
    CC    VARCHAR2 (3 BYTE),
    DD    VARCHAR2 (12 BYTE)
)
ORGANIZATION EXTERNAL
    (
        TYPE ORACLE_LOADER
        DEFAULT DIRECTORY TEST_DIRECTORY
        ACCESS PARAMETERS (
            RECORDS DELIMITED BY NEWLINE
            BADFILE TEST_DIRECTORY : 'TEST.bad'
            LOGFILE TEST_DIRECTORY : 'TEST.log'
            FIELDS
                TERMINATED BY '\t' LTRIM REJECT ROWS WITH ALL NULL FIELDS
            (AA,
             BB,
             CC,
             DD))
        LOCATION (TEST_DIRECTORY:'TEST.dat'))
    REJECT LIMIT UNLIMITED;

测试数据(用制表符替换^I):

NAME1^I0^I ^IUK
NAME2^I0^I ^IUS

当我删除 LTRIM 时,所有数据都在新数据库上读取(但我们需要保留 LTRIM,因为输入文件包含不必要的 spaces)。我注意到一个字段的值为 1 space,它看起来是导致该问题的原因,但为什么只在新数据库上出现?任何想法是什么原因或如何轻松解决?

NLS db/session 参数在两个数据库上相同...但也许有一些全局参数可能导致此问题?

手动更新的测试数据在两个数据库上都有效(用 X 替换第三列中的白色space)

NAME1^I0^IX^IUK
NAME2^I0^IX^IUS

演示:

以下 table 创建于 11g 和 19c:

CREATE TABLE TEST
(
    AA    VARCHAR2 (40 BYTE),
    BB    VARCHAR2 (2 BYTE),
    CC    VARCHAR2 (3 BYTE),
    DD    VARCHAR2 (12 BYTE)
)
ORGANIZATION EXTERNAL
    (
        TYPE ORACLE_LOADER
        DEFAULT DIRECTORY TEST_DIRECTORY
        ACCESS PARAMETERS (
            RECORDS DELIMITED BY NEWLINE
            BADFILE TEST_DIRECTORY : 'TEST.bad'
            LOGFILE TEST_DIRECTORY : 'TEST.log'
            FIELDS 
                TERMINATED BY '\t' LTRIM
                  REJECT ROWS WITH ALL NULL FIELDS
            (AA,
             BB,
             CC  ,
             DD))
        LOCATION (TEST_DIRECTORY:'TEST.dat'))
    REJECT LIMIT UNLIMITED;

两个 tables 来源相同的文件 TEST.dat(数据由制表符分隔,显示为 2 个字符 ^I):

$ cat -A TEST.dat
NAME1^I0^I ^IUK$
NAME2^I0^I ^IUS$

在 11g 上查询:

SQL> SELECT * FROM TEST;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

SQL> SELECT dump(CC) FROM TEST;

DUMP(CC)
--------------------------------------------------------------------------------
NULL
NULL

在 19c 上查询:

SQL> SELECT * FROM TEST;

no rows selected

TEST.log 在 19c 上 运行 查询后显示:

Bad File: TEST.bad

Field Definitions for table TEST
  Record format DELIMITED BY NEWLINE
  Data in file has same endianness as the platform
  Reject rows with all null fields

  Fields in Data Source:

    AA                              CHAR (255)
      Terminated by "   "
      Trim whitespace from left
    BB                              CHAR (255)
      Terminated by "   "
      Trim whitespace from left
    CC                              CHAR (255)
      Terminated by "   "
      Trim whitespace from left
    DD                              CHAR (255)
      Terminated by "   "
      Trim whitespace from left
KUP-04021: field formatting error for field DD
KUP-04023: field start is after end of record
KUP-04101: record 1 rejected in file /home/fff/TEST.dat
KUP-04021: field formatting error for field DD
KUP-04023: field start is after end of record
KUP-04101: record 2 rejected in file /home/fff/TEST.dat

然后,我在没有 LTRIM 的情况下在两个数据库上重新创建了 tables:

CREATE TABLE TEST
(
    AA    VARCHAR2 (40 BYTE),
    BB    VARCHAR2 (2 BYTE),
    CC    VARCHAR2 (3 BYTE),
    DD    VARCHAR2 (12 BYTE)
)
ORGANIZATION EXTERNAL
    (
        TYPE ORACLE_LOADER
        DEFAULT DIRECTORY TEST_DIRECTORY
        ACCESS PARAMETERS (
            RECORDS DELIMITED BY NEWLINE
            BADFILE TEST_DIRECTORY : 'TEST.bad'
            LOGFILE TEST_DIRECTORY : 'TEST.log'
            FIELDS 
                TERMINATED BY '\t'
                  REJECT ROWS WITH ALL NULL FIELDS
            (AA,
             BB,
             CC  ,
             DD))
        LOCATION (TEST_DIRECTORY:'TEST.dat'))
    REJECT LIMIT UNLIMITED;

查询 11g 中的新 table:

SQL> SELECT * FROM TEST;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

SQL> SELECT dump(CC) FROM TEST;

DUMP(CC)
--------------------------------------------------------------------------------
Typ=1 Len=1: 32
Typ=1 Len=1: 32

查询 19c 中的新 table:

SQL> SELECT * FROM TEST;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

SQL> SELECT dump(CC) FROM TEST;

DUMP(CC)
--------------------------------------------------------------------------------
Typ=1 Len=1: 32
Typ=1 Len=1: 32

它不是 LTRIM,它是 LDRTRIM。

SQL> create table et
  2  ( c1 varchar2(16),
  3    c2 varchar2(8),
  4    c3 varchar2(8),
  5    c4 varchar2(8),
  6    c5 varchar2(8),
  7    c6 varchar2(8),
  8    c7 varchar2(8)
  9  )
 10  ORGANIZATION EXTERNAL
 11    (  TYPE ORACLE_LOADER
 12       DEFAULT DIRECTORY temp
 13       ACCESS PARAMETERS
 14         ( RECORDS DELIMITED BY NEWLINE
 15          BADFILE temp: 'TEST_FILE.bad'
 16          LOGFILE temp: 'TEST_FILE.log'
 17          FIELDS TERMINATED BY X'20A7' LTRIM
 18          REJECT ROWS WITH ALL NULL FIELDS
 19         (
 20  c1,c2,c3,c4,c5,c6,c7
 21  )                   )
 22       LOCATION (temp:'TEST_FILE.dat')
 23    )
 24  REJECT LIMIT UNLIMITED;

Table created.

SQL>
SQL> select * from et;

C1               C2       C3       C4       C5       C6       C7
---------------- -------- -------- -------- -------- -------- --------
31234569999999   0        A        X        0        Z        GGGG

SQL>
SQL> drop table et;

Table dropped.

SQL>
SQL> create table et
  2  ( c1 varchar2(16),
  3    c2 varchar2(8),
  4    c3 varchar2(8),
  5    c4 varchar2(8),
  6    c5 varchar2(8),
  7    c6 varchar2(8),
  8    c7 varchar2(8)
  9  )
 10  ORGANIZATION EXTERNAL
 11    (  TYPE ORACLE_LOADER
 12       DEFAULT DIRECTORY temp
 13       ACCESS PARAMETERS
 14         ( RECORDS DELIMITED BY NEWLINE
 15          BADFILE temp: 'TEST_FILE.bad'
 16          LOGFILE temp: 'TEST_FILE.log'
 17          FIELDS TERMINATED BY X'20A7' LDRTRIM
 18          REJECT ROWS WITH ALL NULL FIELDS
 19         (
 20  c1,c2,c3,c4,c5,c6,c7
 21  )                   )
 22       LOCATION (temp:'TEST_FILE.dat')
 23    )
 24  REJECT LIMIT UNLIMITED;

Table created.

SQL>
SQL> select * from et;

C1               C2       C3       C4       C5       C6       C7
---------------- -------- -------- -------- -------- -------- --------
 31234569999999  0        A        X        0                 GGGG
 31234569999999  0        A        X        0        Z        GGGG

让我尝试在我自己的环境中重现您的问题

在 Red Hat Linux 7.2

上使用 Oracle 19c
SQL> select version from v$instance ;

VERSION
-----------------
19.0.0.0.0

演示

更新:分隔符是制表符

文件内容

$ cat -A TEST.dat
NAME1^I0^I ^IUK$
NAME2^I0^I ^IUS$

外部Table

SQL> drop table TEST_EXTERNAL_TABLE ;

Table dropped.

SQL> CREATE TABLE TEST_EXTERNAL_TABLE
  2  (
  3      AA    VARCHAR2 (40 BYTE),
  4      BB    VARCHAR2 (2 BYTE),
  5      CC    VARCHAR2 (3 BYTE),
  6      DD    VARCHAR2 (12 BYTE)
  7  )
  8  ORGANIZATION EXTERNAL
  9      (
 10          TYPE ORACLE_LOADER
 11          DEFAULT DIRECTORY DIR_TEST
 12          ACCESS PARAMETERS (
 13              RECORDS DELIMITED BY NEWLINE
 14              BADFILE DIR_TEST : 'TEST.bad'
 15              LOGFILE DIR_TEST : 'TEST.log'
 16              FIELDS TERMINATED BY '\t' NOTRIM
 17                     REJECT ROWS WITH ALL NULL FIELDS
 18              (AA,
 19               BB,
 20               CC,
 21               DD))
 22*         LOCATION (DIR_TEST:'TEST.dat'))
SQL> /

Table created.

SQL>  select * from TEST_EXTERNAL_TABLE ;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

SQL> select dump(cc) from TEST_EXTERNAL_TABLE ;

DUMP(CC)
--------------------------------------------------------------------------------
Typ=1 Len=1: 32
Typ=1 Len=1: 32

在我的例子中,我能够加载,但空白区域仍然存在,这是 NOTRIMLDRTRIM.

的预期行为

LDRTRIM is used to provide compatibility with SQL*Loader trim features. It is the same as NOTRIM except in the following cases:

If the field is not a delimited field, then spaces will be trimmed from the right. If the field is a delimited field with OPTIONALLY ENCLOSED BY specified, and the optional enclosures are missing for a particular instance, then spaces will be trimmed from the left.

LDRTRIM

做同样的事情
SQL> drop table TEST_eXTERNAL_TABLE;

Table dropped.

SQL> l
  1  CREATE TABLE TEST_EXTERNAL_TABLE
  2  (
  3      AA    VARCHAR2 (40 BYTE),
  4      BB    VARCHAR2 (2 BYTE),
  5      CC    VARCHAR2 (3 BYTE),
  6      DD    VARCHAR2 (12 BYTE)
  7  )
  8  ORGANIZATION EXTERNAL
  9      (
 10          TYPE ORACLE_LOADER
 11          DEFAULT DIRECTORY DIR_TEST
 12          ACCESS PARAMETERS (
 13              RECORDS DELIMITED BY NEWLINE
 14              BADFILE DIR_TEST : 'TEST.bad'
 15              LOGFILE DIR_TEST : 'TEST.log'
 16              FIELDS TERMINATED BY '\t' LDRTRIM
 17                     REJECT ROWS WITH ALL NULL FIELDS
 18              (AA,
 19               BB,
 20               CC,
 21               DD))
 22*         LOCATION (DIR_TEST:'TEST.dat'))
SQL> /

Table created.

SQL> select * from TEST_EXTERNAL_TABLE ;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

SQL> select dump(cc) from TEST_EXTERNAL_TABLE ;

DUMP(CC)
--------------------------------------------------------------------------------
Typ=1 Len=1: 32
Typ=1 Len=1: 32

SQL>

如果您使用 LTRIM 它不起作用,因为空格在右侧,因为该字段是空的。这是默认行为,至少因为 12c 是它的工作方式并且应该是这样。

SQL> drop table TEST_EXTERNAL_TABLE ;

Table dropped.

SQL> CREATE TABLE TEST_EXTERNAL_TABLE
(
    AA    VARCHAR2 (40 BYTE),
  2    3    4      BB    VARCHAR2 (2 BYTE),
    CC    VARCHAR2 (3 BYTE),
  5    6      DD    VARCHAR2 (12 BYTE)
  7  )
  8  ORGANIZATION EXTERNAL
    (
  9   10          TYPE ORACLE_LOADER
        DEFAULT DIRECTORY DIR_TEST
        ACCESS PARAMETERS (
 11   12   13              RECORDS DELIMITED BY NEWLINE
            BADFILE DIR_TEST : 'TEST.bad'
            LOGFILE DIR_TEST : 'TEST.log'
 14   15   16              FIELDS TERMINATED BY '\t' LTRIM
                        REJECT ROWS WITH ALL NULL FIELDS
            (AA,
             BB,
 17   18   19   20               CC,
             DD))
        LOCATION (DIR_TEST:'TEST.dat'))
 21   22   23      REJECT LIMIT UNLIMITED;

Table created.

SQL> select * from TEST_EXTERNAL_TABLE ;

no rows selected

现在 RTRIM 可以按预期工作,因为整个字段中的空格是从右到左处理的。

SQL> drop table TEST_EXTERNAL_TABLE ;

Table dropped.

SQL> CREATE TABLE TEST_EXTERNAL_TABLE
  2  (
    AA    VARCHAR2 (40 BYTE),
  3    4      BB    VARCHAR2 (2 BYTE),
    CC    VARCHAR2 (3 BYTE),
    DD    VARCHAR2 (12 BYTE)
  5    6    7  )
ORGANIZATION EXTERNAL
    (
  8    9   10          TYPE ORACLE_LOADER
 11          DEFAULT DIRECTORY DIR_TEST
        ACCESS PARAMETERS (
            RECORDS DELIMITED BY NEWLINE
 12   13   14              BADFILE DIR_TEST : 'TEST.bad'
            LOGFILE DIR_TEST : 'TEST.log'
 15   16              FIELDS TERMINATED BY '\t' RTRIM
 17                     REJECT ROWS WITH ALL NULL FIELDS
 18              (AA,
       19         BB,
 20               CC,
             DD))
        LOCATION (DIR_TEST:'TEST.dat'))
 21   22   23      REJECT LIMIT UNLIMITED;

Table created.

SQL> select * from TEST_EXTERNAL_TABLE ;

AA                                       BB CC  DD
---------------------------------------- -- --- ------------
NAME1                                    0      UK
NAME2                                    0      US

我的建议:使用 LDRTRIM,或者更好的是,避免所有空格都是一种选择。关于您在 11g 中的测试,那是一个相当旧的版本,并且该行为可能是错误的结果,尽管我找不到任何报告来解释此行为。