Python

Question

我正在尝试创建同时从列表中进行运行多个查询的方法，因为我正在使用线程库。为此，我有这个代码：

from threading import Thread, Lock
queries = ["SELECT * FROM db1.trans", "SELECT * FROM db1.order", "SELECT * FROM db2.Store", "SELECT * FROM db2.Document", "SELECT * FROM db3.Sales"]
class DatabaseWorker(Thread):
    __lock = Lock()

    def __init__(self, query, result_queue):
        Thread.__init__(self)
        self.query = query
        self.result_queue = result_queue

    def run(self):
        result = None
        print("Connecting to database...")
        try:
            conn = connect(host=host, port=port)
            curs = conn.cursor()
            curs.execute(self.query)
            result = curs
            curs.close()
            conn.close()
        except Exception as e:
            print(str(e))
        self.result_queue.append(result)

delay = 1
result_queue = []
for query in queries:
    worker1 = DatabaseWorker(query,result_queue)

worker1.start()
while len(result_queue) < 2:
    time.sleep(delay)
job_done = True
worker1.join()

使用上述方法，我运行在顺序模式下。我知道我可以这样做：

worker1 = DatabaseWorker(queries[0],result_queue)
worker2 = DatabaseWorker(queries[1],result_queue)
...

但我认为这不是最好的方法。任何人都知道如何在动态模式下运行列表中的所有查询？

谢谢！

Answer 1

Python 由于 Python GIL（全局解释器锁），线程并不是真正的并行。

对于真正的多处理并行操作，您可以使用 Python 多处理模块。

示例：

import multiprocessing

def runner(task):
    return f'Hi, i do {task}'


if __name__ == '__main__':
    list_tasks = ['1', '2', '3']
    with multiprocessing.Pool() as pool:
        result = pool.map(runner, list_tasks)
    print(result)

Python - 基于列表的动态并行处理

Python - Dynamic processing in parallel based on a list

multithreading

python-multiprocessing