如何使用带有 Sybase 的 SQLAlchemy 1.3.18 来实现 fast_executemany 之类的东西来提高插入速度?

How to have something like fast_executemany with SQLAlchemy 1.3.18 with Sybase to improve the speed of insert into?

我在 Python 中制作了一个插入 Sybase 数据库的程序,我使用 pandas 1.0.4 和 SQLAlchemy 1.3.18。

我已经在sybase和我的脚本之间做了link,但实际上插入表的速度真的很慢(30k行需要10分钟...)

在SQL服务器中,代码相同(除了create_engine中参数fast_executemany=True,参数method='multi',chunksize = 500 for dataframe.to_sql()), 我把插入变成了 3 secondes.

你有什么办法解决这个问题吗?

祝你有愉快的一天!

看起来 Sybase ASE does not support a multiple values VALUES clause in INSERT. You could form a UNION of SELECT statements, if you need to insert multiple values in a single statement instead of using executemany 只是发出许多 INSERT 语句:

from sqlalchemy import union_all, select, literal

def sybase_insert(sqltable, conn, keys, data_iter):
    sel = union_all(*[select([literal(v) for v in row]) for row in data_iter])
    ins = sqltable.table.insert().from_select(keys, sel)
    conn.execute(ins)

调用to_sql()时将函数作为method传递:

In [11]: pd.DataFrame({"A": range(3)}).to_sql("foo", con=engine, method=sybase_insert)
2020-08-10 08:29:44,474 INFO sqlalchemy.engine.base.Engine PRAGMA main.table_info("foo")
2020-08-10 08:29:44,474 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,475 INFO sqlalchemy.engine.base.Engine PRAGMA temp.table_info("foo")
2020-08-10 08:29:44,475 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine 
CREATE TABLE foo (
        "index" BIGINT, 
        "A" BIGINT
)


2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine COMMIT
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine CREATE INDEX ix_foo_index ON foo ("index")
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine COMMIT
2020-08-10 08:29:44,479 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine INSERT INTO foo ("index", "A") SELECT ? AS anon_1, ? AS anon_2 UNION ALL SELECT ? AS anon_3, ? AS anon_4 UNION ALL SELECT ? AS anon_5, ? AS anon_6
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine (0, 0, 1, 1, 2, 2)
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine COMMIT

如果您使用 SAP ASE ODBC 驱动程序,external SAP ASE (Sybase) dialect 确实支持 fast_executemany