Python - 基于列表的动态并行处理
Python - Dynamic processing in parallel based on a list
我正在尝试创建同时从列表中进行 运行 多个查询的方法,因为我正在使用线程库。为此,我有这个代码:
from threading import Thread, Lock
queries = ["SELECT * FROM db1.trans", "SELECT * FROM db1.order", "SELECT * FROM db2.Store", "SELECT * FROM db2.Document", "SELECT * FROM db3.Sales"]
class DatabaseWorker(Thread):
__lock = Lock()
def __init__(self, query, result_queue):
Thread.__init__(self)
self.query = query
self.result_queue = result_queue
def run(self):
result = None
print("Connecting to database...")
try:
conn = connect(host=host, port=port)
curs = conn.cursor()
curs.execute(self.query)
result = curs
curs.close()
conn.close()
except Exception as e:
print(str(e))
self.result_queue.append(result)
delay = 1
result_queue = []
for query in queries:
worker1 = DatabaseWorker(query,result_queue)
worker1.start()
while len(result_queue) < 2:
time.sleep(delay)
job_done = True
worker1.join()
使用上述方法,我 运行 在顺序模式下。我知道我可以这样做:
worker1 = DatabaseWorker(queries[0],result_queue)
worker2 = DatabaseWorker(queries[1],result_queue)
...
但我认为这不是最好的方法。任何人都知道如何在动态模式下 运行 列表中的所有查询?
谢谢!
Python 由于 Python GIL(全局解释器锁),线程并不是真正的并行。
对于真正的多处理并行操作,您可以使用 Python 多处理模块。
示例:
import multiprocessing
def runner(task):
return f'Hi, i do {task}'
if __name__ == '__main__':
list_tasks = ['1', '2', '3']
with multiprocessing.Pool() as pool:
result = pool.map(runner, list_tasks)
print(result)
我正在尝试创建同时从列表中进行 运行 多个查询的方法,因为我正在使用线程库。为此,我有这个代码:
from threading import Thread, Lock
queries = ["SELECT * FROM db1.trans", "SELECT * FROM db1.order", "SELECT * FROM db2.Store", "SELECT * FROM db2.Document", "SELECT * FROM db3.Sales"]
class DatabaseWorker(Thread):
__lock = Lock()
def __init__(self, query, result_queue):
Thread.__init__(self)
self.query = query
self.result_queue = result_queue
def run(self):
result = None
print("Connecting to database...")
try:
conn = connect(host=host, port=port)
curs = conn.cursor()
curs.execute(self.query)
result = curs
curs.close()
conn.close()
except Exception as e:
print(str(e))
self.result_queue.append(result)
delay = 1
result_queue = []
for query in queries:
worker1 = DatabaseWorker(query,result_queue)
worker1.start()
while len(result_queue) < 2:
time.sleep(delay)
job_done = True
worker1.join()
使用上述方法,我 运行 在顺序模式下。我知道我可以这样做:
worker1 = DatabaseWorker(queries[0],result_queue)
worker2 = DatabaseWorker(queries[1],result_queue)
...
但我认为这不是最好的方法。任何人都知道如何在动态模式下 运行 列表中的所有查询?
谢谢!
Python 由于 Python GIL(全局解释器锁),线程并不是真正的并行。
对于真正的多处理并行操作,您可以使用 Python 多处理模块。
示例:
import multiprocessing
def runner(task):
return f'Hi, i do {task}'
if __name__ == '__main__':
list_tasks = ['1', '2', '3']
with multiprocessing.Pool() as pool:
result = pool.map(runner, list_tasks)
print(result)