COPY-FROM within python connection.execute: csv into postgres - 没有错误但也没有数据

Question

我正在尝试将数据从 csv 加载到我的 postgres database.I 我正在使用 sqlalchemy（和连接/原始 sql 数据方法而不是 ORM 方法）。但是，我使用相应字段成功创建了 tables。使用 copy-from 时没有任何反应：我没有收到错误，但我的 table 也是空的。

conn = eng.connect()
trans = conn.begin()

try: 
    conn.execute("""CREATE TABLE IF NOT EXISTS table_name(var1 numeric, date date, time time, datetime timestamp primary key,....);""")
    trans.commit()
except:
    trans.rollback()
    raise

try: 
    # File 1 bid
    conn.execute("""COPY table_name FROM '/home/user/csvfile.csv' 
    WITH CSV HEADER DELIMITER as ','""")
    trans.commit()
except:
    trans.rollback()
    raise

Jupyter notebook 没有报告任何错误消息。

我做错了什么？

另外，我不是很懂：

file = "/home/user/csvfile.csv"
  conn.execute("""COPY table_name FROM file 
        WITH CSV HEADER DELIMITER as ','""")

在 "file" 中产生错误。

Answer 1

尝试添加“;”到您的复制命令。 ... 分隔符 ',' CSV ;" :-)

Answer 2

事务不可重用：

>>> trans = conn.begin()
2018-03-29 09:14:33,001 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
>>> trans.commit()
2018-03-29 09:14:35,449 INFO sqlalchemy.engine.base.Engine COMMIT
>>> trans.commit()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/u/sqlalchemy/lib/sqlalchemy/engine/base.py", line 1642, in commit
    raise exc.InvalidRequestError("This transaction is inactive")
sqlalchemy.exc.InvalidRequestError: This transaction is inactive
>>> trans.rollback()
>>>

您必须在第一个事务之后开始一个新事务，或者在同一个事务中执行所有操作。

另一方面，这并不能解释您的 COPY 失败的原因。在前一个显式事务结束后，连接回退到 autocommit. Put another way an engine or a connection commits if no transaction is in progress. But here's the catch: autocommit is based on detecting data changing operations, which is done by matching the given statement 反对

AUTOCOMMIT_REGEXP = re.compile(
    r'\s*(?:UPDATE|INSERT|CREATE|DELETE|DROP|ALTER|GRANT|REVOKE|'
    'IMPORT FOREIGN SCHEMA|REFRESH MATERIALIZED VIEW|TRUNCATE)',
    re.I | re.UNICODE)

您可能会注意到，COPY 不是该正则表达式的一部分。如前所述，最好的办法是启动另一个显式事务，或者在同一个事务中执行这两个操作。但是如果你希望在将来的某个时间通过 COPY 使用自动提交，请指示 SQLAlchemy 它应该自动提交：

conn.execute(text("COPY ...").execution_options(autocommit=True))

至于后一个错误，FROM file并没有神奇地访问Python变量，属于语法错误。将文件名作为参数传递给查询：

copy_stmt = text("COPY table_name FROM :file WITH CSV HEADER")
copy_stmt = copy_stmt.execution_options(autocommit=True)
conn.execute(copy_stmt, {"file": file})

请注意，COPY from a file 需要您可能不应该拥有的特权：

COPY naming a file or command is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access.

解决方案是使用COPY ... FROM STDIN，但是要使用那个:

file = "/home/user/csvfile.csv"
stmt = "COPY table_name FROM STDIN CSV HEADER"

raw_conn = eng.raw_connection()

# Uses the actual psycopg2 connection as a context manager
# for transaction handling.
with open(file) as f, \
        raw_conn.connection, \
        raw_conn.cursor() as cur:
    cur.copy_expert(stmt, f)

raw_conn.close()

COPY-FROM within python connection.execute: csv into postgres - 没有错误但也没有数据

COPY-FROM within python connection.execute: csv into postgres - no error but also no data

python

csv

postgresql

connection

sqlalchemy