如何使用 Python 多线程处理 MySQL 连接

Question

我有一个主要的 Python 脚本，它连接到 MySQL 数据库并从中提取一些记录。根据返回的结果，它会启动与抓取的记录一样多的线程（class 个实例）。每个线程都应返回数据库并通过将一个状态标志设置为不同的状态 ("process started") 来更新另一个 table。

为了实现这一点，我尝试了：

1.) 将数据库连接传递给所有线程 2.) 从每个线程打开一个新的数据库连接

但其中 none 正在工作。

通过使用 try/except，我可以运行我的更新在这两种情况下都没有任何问题，但是 MySQL table 尚未更新，并且没有生成错误.我在这两种情况下都使用了提交。

我的问题是在这种情况下如何处理 MySQL 个连接？

根据前几条评论更新：

MAIN SCRIPT
-----------

#Connecting to DB
db = MySQLdb.connect(host = db_host,
                         db = db_db,
                         port = db_port,
                         user = db_user,
                         passwd = db_password,
                         charset='utf8')

# Initiating database cursor
cur = db.cursor()

# Fetching records for which I need to initiate a class instance

cur.execute('SELECT ...')

for row in cur.fetchall() :
    # Initiating new instance, appending it to a list and
    # starting all of them 



CLASS WHICH IS INSTANTIATED
---------------------------

# Connecting to DB again. I also tried to pass connection
# which has been opened in the main script but it did not
# work either.

db = MySQLdb.connect(host = db_host,
                         db = db_db,
                         port = db_port,
                         user = db_user,
                         passwd = db_password,
                         charset='utf8')

# Initiating database cursor
cur_class = db.cursor()
cur.execute('UPDATE ...')
db.commit()

Answer 1

这里是在Python中使用多线程处理mysql的例子，我不知道您的 table 和数据，所以，只需更改代码可能会有所帮助：

import threading
import time
import MySQLdb

Num_Of_threads = 5

class myThread(threading.Thread):

    def __init__(self, conn, cur, data_to_deal):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.conn = conn
        self.cur = cur
        self.data_to_deal

    def run(self):

        # add your sql 
        sql = 'insert into table id values ({0});'
        for i in self.data_to_deal:
            self.cur.execute(sql.format(i))
            self.conn.commit()

threads = []
data_list = [1,2,3,4,5]

for i in range(Num_Of_threads):
    conn = MySQLdb.connect(host='localhost',user='root',passwd='',db='')
    cur = conn.cursor()
    new_thread = myThread(conn, cur, data_list[i])

for th in threads:
    th.start()

for t in threads:
    t.join()

Answer 2

看来我的代码没有问题，但我的 MySQL 版本。我正在使用 MySQL 标准社区版，并基于发现的官方文档 here :

The thread pool plugin is a commercial feature. It is not included in MySQL community distributions.

我准备升级到 MariaDB 来解决这个问题。

Answer 3

看起来 mysql 5.7 确实支持多线程。

正如您之前尝试的那样 - 绝对确保在 def worker() 中传递连接。全局定义连接是我的错误

下面是通过 5 个线程打印 10 条记录 5 次的示例代码

import MySQLdb
import threading


def write_good_proxies():    
    local_db = MySQLdb.connect("localhost","username","PassW","DB", port=3306 )
    local_cursor = local_db.cursor (MySQLdb.cursors.DictCursor)
    sql_select = 'select http from zproxies where update_time is null order by rand() limit 10'
    local_cursor.execute(sql_select)
    records = local_cursor.fetchall()
    id_list = [f['http'] for f in records]
    print id_list
def worker():
    x=0
    while x< 5:
        x = x+1
        write_good_proxies()

threads = []


for i in range(5):
    print i
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()

如何使用 Python 多线程处理 MySQL 连接

How to handle MySQL connection(s) with Python multithreading

python

mysql

multithreading