模块属性更新不会传播到 Windows 上的子进程

Question

我有一些与 Windows 上的模块属性更新相关的问题没有传播到 Windows 上的子进程。

以下片段说明了问题：

import functools
import multiprocessing
import os
from contextlib import contextmanager

_DOMAIN_RANGE_SCALE = 'reference'


def get_domain_range_scale():
    return _DOMAIN_RANGE_SCALE


def set_domain_range_scale(scale='Reference'):
    global _DOMAIN_RANGE_SCALE

    scale = str(scale).lower()

    _DOMAIN_RANGE_SCALE = scale


class domain_range_scale(object):
    def __init__(self, scale):
        self._scale = scale
        self._previous_scale = get_domain_range_scale()

    def __enter__(self):
        set_domain_range_scale(self._scale)

        return self

    def __exit__(self, *args):
        set_domain_range_scale(self._previous_scale)

    def __call__(self, function):
        @functools.wraps(function)
        def wrapper(*args, **kwargs):
            with self:
                return function(*args, **kwargs)

        return wrapper


@contextmanager
def multiprocessing_pool(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)

    yield pool

    pool.terminate()


def test_domain_range_scale(*args):
    print('Domain Range Scale Inner: {0}, PID: {1}'.format(
        get_domain_range_scale(), os.getpid()))


if __name__ == '__main__':
    for scale in ('reference', '1', '100'):
        with domain_range_scale(scale):
            print('*' * 79)
            print('Domain Range Scale Outer: {0}, PID: {1}'.format(
                get_domain_range_scale(), os.getpid()))
            with multiprocessing_pool(processes=4) as pool:
                pool.map(test_domain_range_scale, range(10))

在 Linux / macOS

上输出

*******************************************************************************
Domain Range Scale Outer: reference, PID: 93989
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93992
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93992
*******************************************************************************
Domain Range Scale Outer: 1, PID: 93989
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93995
*******************************************************************************
Domain Range Scale Outer: 100, PID: 93989
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999

输出 Windows

*******************************************************************************
Domain Range Scale Outer: reference, PID: 6524
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 5476
Domain Range Scale Inner: reference, PID: 4872
Domain Range Scale Inner: reference, PID: 1932
*******************************************************************************
Domain Range Scale Outer: 1, PID: 6524
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 1012
Domain Range Scale Inner: reference, PID: 1852
Domain Range Scale Inner: reference, PID: 6544
*******************************************************************************
Domain Range Scale Outer: 100, PID: 6524
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 5944

Answer 1

您的问题在于 Windows 不支持 "fork" 作为新进程的启动方法（仅 "spawn"）。 "spawn" 不继承全局变量。当你在 _DOMAIN_RANGE_SCALE = 'reference' 下面放置一个打印语句时，你会看到 Windows 上的子进程将再次运行脚本直到 if __name__ == '__main__': 他们导入所需的功能。

您必须使用 Pool 的 initializer 参数在进程启动后显式注册全局变量。

...

def init_global(scale):
    global _DOMAIN_RANGE_SCALE
    _DOMAIN_RANGE_SCALE = scale

if __name__ == '__main__':

...
        with multiprocessing.Pool(processes=4,
                                  initializer=init_global,
                                  initargs=(scale,)) as pool:

            pool.map(test_domain_range_scale, range(10))
...

模块属性更新不会传播到 Windows 上的子进程

Module attribute update is not propagated to child processes on Windows

python

multiprocessing

contextmanager