如何使用python批量提交数据到SQLServer数据库？

Question

我要插入数据库的数据是这样的：

datalist =[['2012', '1', '3', '1', '832.0', '261.0', '100.00'],
            ['2012', '1', '5', '1', '507.0', '193.0', '92.50'],
            ['2012', '2', '3', '1', '412.0', '200.0', '95.00'],
            ['2012', '2', '5', '1', '560.0', '335.0', '90.00'],
            ['2012', '3', '3', '1', '584.0', '205.0', '100.00'],
            ['2012', '3', '5', '1', '595.0', '162.0', '92.50'],
            ['2012', '4', '3', '1', '504.0', '227.0', '100.00'],
            ['2012', '4', '5', '1', '591.0', '264.0', '92.50']]

但实际上datalist中有50万行。所以我只列出了一部分。

我插入数据库的代码是这样的：

import pymssql

server = '127.0.0.1'
user = "test"
password = "test"
database='SQLTest'
datalist = [['2012', '1', '3', '1', '832.0', '261.0', '100.00'],
            ['2012', '1', '5', '1', '507.0', '193.0', '92.50'],
            ['2012', '2', '3', '1', '412.0', '200.0', '95.00'],
            ['2012', '2', '5', '1', '560.0', '335.0', '90.00'],
            ['2012', '3', '3', '1', '584.0', '205.0', '100.00'],
            ['2012', '3', '5', '1', '595.0', '162.0', '92.50'],
            ['2012', '4', '3', '1', '504.0', '227.0', '100.00'],
            ['2012', '4', '5', '1', '591.0', '264.0', '92.50']]

#But in fact, there are 500,000 rows in datalist

try:
    conn = pymssql.connect(server, user, password, database)
    cursor = conn.cursor()
    for one_row in datalist:
        val1 = one_row[4]
        val2 = one_row[5]
        val3 = one_row[6]
        sql = "insert into table_for_test values(col1, col2, col3)" % (val1, val2,val3)
        cursor.execute(sql)
        conn.commit()
except Exception as ex:
    conn.rollback()
    raise ex
finally:
    conn.close()

因为数据量太大，所以想批量插入数据，如何修改代码？

Answer 1

执行此操作的一种方法是使用 BULK INSERT 语句。 https://docs.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql

输入应来自文件（例如 CSV）。

例如，如果数据在 CSV 文件中

BULK INSERT table_for_test
    FROM C:\user\admin\downloads\mycsv.csv
    WITH (
        FIRSTROW=1
      , FIELDTERMINATOR=','
      , ROWTERMINATOR='\n'
    )

Answer 2

现在我知道怎么做了。使用executeMany。元素必须是列表中的元组。

import pymssql

server = '127.0.0.1'
user = "test"
password = "test"
database='SQLTest'
datalist = [('2012', '1', '3', '1', '832.0', '261.0', '100.00'),
            ('2012', '1', '5', '1', '507.0', '193.0', '92.50'),
            ('2012', '2', '3', '1', '412.0', '200.0', '95.00'),
            ('2012', '2', '5', '1', '560.0', '335.0', '90.00'),
            ('2012', '3', '3', '1', '584.0', '205.0', '100.00'),
            ('2012', '3', '5', '1', '595.0', '162.0', '92.50'),
            ('2012', '4', '3', '1', '504.0', '227.0', '100.00'),
            ('2012', '4', '5', '1', '591.0', '264.0', '92.50')]

try:
    conn = pymssql.connect(server, user, password, database)
    cursor = conn.cursor()
    sql = "insert into table_for_test values(col1, col2, col3, col4, col5, col6, col7) values(%s, %s, %s, %s, %s, %s, %s)"
    cursor.executemany(sql, datalist)
    conn.commit()
except Exception as ex:
    conn.rollback()
    raise ex
finally:
    conn.close()

如何使用python批量提交数据到SQLServer数据库？

How to use python to commit data in batches to SQLServer database?

python

pymssql