Psycopg2 copy_from csv 忽略行中的逗号

Question

我正在尝试使用 psycopg2 的 copy_from 方法将 csv 文件复制到 postgres table (phsyical)。

columns = (
    'upc',
    'date',
    'type',
    'country_code',
    'customer'
    )

def insert_csv(f, columns):
    connection = get_postgres_connection()
    cursor = connection.cursor()
    cursor.copy_from(f, 'physical', sep=',', columns=columns)
    connection.commmit()

我为未包含在 columns 中的每一行自动生成了一个 ID。

但是，某些行的单元格内有逗号。进行故障排除时，我可以看到有错误的行：

1111111,2021-02-28 00:00:00,,US,"Name, The"

我找到了这个 Whosebug question 并尝试了：

cursor.copy_expert("COPY physical FROM STDIN WITH (FORMAT CSV)", f)

但这会导致我之前运行在未指定我想要的列时遇到的错误。

value "1111111" is out of range for type integer
CONTEXT:  COPY physical, line 1, column id: "1111111"

有谁知道解决这个问题的方法吗？

更新的工作代码：

def insert_csv(f, table, columns):
    connection = get_postgres_connection()
    cursor = connection.cursor()
    try:
        column_names = ','.join(columns)
        query = f'''
            COPY {table}({column_names})
            FROM STDOUT (FORMAT CSV)
        '''
        cursor.copy_expert(query, f)
        connection.commit()
        return True
    except (psycopg2.Error) as e:
        print(e)
        return False
    finally:
        cursor.close()
        connection.close()

columns = (
        "upc",
        "date_thru",
        "transaction_type",
        "transaction_type_subtype",
        "country_code",
        "customer",
        "quantity",
        "income_gross",
        "fm_serial",
        "date_usage"
    )

with open(dump_file, 'r', newline='', encoding="ISO-8859-1") as f:
        inserted = insert_csv(f, 'physical', columns)

Answer 1

在复制命令中指定列名，例如：

column_names = ','.join(columns)
copy_cmd = f"copy physical({column_names}) from stdout (format csv)"
cursor.copy_expert(copy_cmd, f)

Psycopg2 copy_from csv 忽略行中的逗号

Psycopg2 copy_from csv ignore commas in row

python

postgresql

copy

psycopg2