multiprocessing.Queue 分叉时的行为

Question

os.fork() 之后，我正在尝试与 children 交换数据。为此，我使用 multiprocessing.Queue 个实例。当 parent put 和 children get 时，队列正常工作；但不是相反。

我的示例代码：

import os
import multiprocessing as mp
from queue import Empty

if __name__ == '__main__':

    n_workers = 5

    forward_queue = mp.Queue()
    pids_queue = mp.Queue()

    for n in range(n_workers):
        forward_queue.put(n)

    for n in range(n_workers):
        child = os.fork()
        if child:
            pass
        else:
            my_number = forward_queue.get()
            print('pid={} here, my number is {}'.format(os.getpid(), my_number))
            pids_queue.put(os.getpid())
            os._exit(0)  # correct way to exit a fork according to docs

    while True:
        try:
            pid_of_child = pids_queue.get(timeout=5)
        except Empty:
            print('no more pids')
            break
        else:
            print('one of my children had this pid={}'.format(pid_of_child))

我得到的输出：

pid=19715 here, my number is 0
pid=19716 here, my number is 1
pid=19717 here, my number is 2
pid=19721 here, my number is 3
pid=19718 here, my number is 4
no more pids

我期望的输出：

pid=19715 here, my number is 0
pid=19716 here, my number is 1
pid=19717 here, my number is 2
pid=19721 here, my number is 3
pid=19718 here, my number is 4
one of my children had this pid=19715
one of my children had this pid=19716
one of my children had this pid=19717
one of my children had this pid=19721
one of my children had this pid=19718
no more pids

有人可以解释为什么会这样吗？

Answer 1

在退出 fork 之前试试这个：

pids_queue.close()
pids_queue.join_thread()

问题是队列是如何工作的。将值放入队列后，后台线程将启动以将项目传输到管道中。当您立即调用 os._exit 时，线程将被关闭。针对这类问题，开发了方法.close和.join_thread。

multiprocessing.Queue 分叉时的行为

multiprocessing.Queue behavior when forking

python

queue

fork

python-multiprocessing

python-os