如何使用带有 Sybase 的 SQLAlchemy 1.3.18 来实现 fast_executemany 之类的东西来提高插入速度?
How to have something like fast_executemany with SQLAlchemy 1.3.18 with Sybase to improve the speed of insert into?
我在 Python 中制作了一个插入 Sybase 数据库的程序,我使用 pandas 1.0.4 和 SQLAlchemy 1.3.18。
我已经在sybase和我的脚本之间做了link,但实际上插入表的速度真的很慢(30k行需要10分钟...)
在SQL服务器中,代码相同(除了create_engine中参数fast_executemany=True,参数method='multi',chunksize = 500 for dataframe.to_sql()), 我把插入变成了 3 secondes.
你有什么办法解决这个问题吗?
祝你有愉快的一天!
看起来 Sybase ASE does not support a multiple values VALUES
clause in INSERT
. You could form a UNION
of SELECT
statements, if you need to insert multiple values in a single statement instead of using executemany
只是发出许多 INSERT
语句:
from sqlalchemy import union_all, select, literal
def sybase_insert(sqltable, conn, keys, data_iter):
sel = union_all(*[select([literal(v) for v in row]) for row in data_iter])
ins = sqltable.table.insert().from_select(keys, sel)
conn.execute(ins)
调用to_sql()
时将函数作为method
传递:
In [11]: pd.DataFrame({"A": range(3)}).to_sql("foo", con=engine, method=sybase_insert)
2020-08-10 08:29:44,474 INFO sqlalchemy.engine.base.Engine PRAGMA main.table_info("foo")
2020-08-10 08:29:44,474 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,475 INFO sqlalchemy.engine.base.Engine PRAGMA temp.table_info("foo")
2020-08-10 08:29:44,475 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine
CREATE TABLE foo (
"index" BIGINT,
"A" BIGINT
)
2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,477 INFO sqlalchemy.engine.base.Engine COMMIT
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine CREATE INDEX ix_foo_index ON foo ("index")
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine ()
2020-08-10 08:29:44,478 INFO sqlalchemy.engine.base.Engine COMMIT
2020-08-10 08:29:44,479 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine INSERT INTO foo ("index", "A") SELECT ? AS anon_1, ? AS anon_2 UNION ALL SELECT ? AS anon_3, ? AS anon_4 UNION ALL SELECT ? AS anon_5, ? AS anon_6
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine (0, 0, 1, 1, 2, 2)
2020-08-10 08:29:44,481 INFO sqlalchemy.engine.base.Engine COMMIT
如果您使用 SAP ASE ODBC 驱动程序,external SAP ASE (Sybase) dialect 确实支持 fast_executemany
。