python 3 池映射不遵循 mac 中的进程指令

Question

我有以下代码

import multiprocessing, subprocess

def runGate(alias):
    #worker function
    command = ['Gate','-a',alias,'main.mac']
    return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

p = multiprocessing.Pool(processes=4)
p.map(runGate, aliases)
p.close()
p.join()

其中别名是一组 1k+ 个组合，显然我不想同时启动 1k+ 个进程，只有 4 个。但事实并非如此，实际上启动了 1k+ 个进程并且我的 mac 重启。我怎样才能一次只启动 4 个进程？

Answer 1

这是实际有效的解决方案

def call_proc(cmd):
    """ This runs in a separate thread. """
    #subprocess.call(shlex.split(cmd))  # This will block until cmd finishes
    p = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out, err = p.communicate()
    return (out, err)

pbar = ProgressBar(widgets=[SimpleProgress()], maxval=len(aliases)).start()

pool = multiprocessing.Pool(processes=6)  #(multiprocessing.cpu_count())
results = []
for alias in aliases:
    command = 'Gate -a '+alias+' main.mac'
    pool.apply_async(call_proc, (command,), callback=results.append)

while len(results) != len(aliases):
    pbar.update(len(results))
    sleep(0.5)
pbar.finish()

# Close the pool and wait for each running task to complete
pool.close()
pool.join()

python 3 池映射不遵循 mac 中的进程指令

python 3 pool map does not follow processes instruction in mac

python-3.x

multiprocessing

pool