Multiprocessing Pool() 方法对性能没有影响

Question

我在 Linux/Debian 测试中使用 Python 3.9.2，在多处理器机器上。我正在尝试了解多处理的工作原理。

我写了两个简单的脚本来执行两个指数函数，一个没有多处理，另一个有。

这是没有多处理的：

from timeit import default_timer as timer


def sqr(n):

    a = n ** n

    return a


def sqr_2(m):

    b = m ** m

    return b


def main():

    start = timer()
    
    print(f'sqr = {sqr(100000)}\nsqr_2= {sqr_2(200000)}')
    
    end = timer()


    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

这是使用多处理的脚本：

from multiprocessing import Pool, cpu_count
from timeit import default_timer as timer


def sqr_1(n):

    return n ** n


def sqr_2(m):

    return m ** m


def main():

    cpu_cnt = cpu_count()
    pool = Pool(processes = cpu_cnt)     #In this case there are 12 processors

    start = timer()
    
    val_1 = (100000,)
    val_2 = (200000,)
    
    process_1 = pool.map_async(sqr_1, val_1)
    process_2 = pool.map_async(sqr_2, val_2)
    
    print(f'Results: {process_1.get(), process_2.get()}')

    end = timer()

    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

问题在于，第二个脚本的进程在没有任何错误的情况下完成，在相同的时间内（大约 14 秒）执行与第一个脚本相同的任务。因此，第二个脚本中的多处理不起作用。我提前感谢任何想指出错误的人！

Answer 1

考虑以下脚本。它允许您在运行时选择调用该函数的次数，以及是串行调用还是并行调用。它也只是计算值；它不会尝试将字符串表示形式写入标准输出（因为将 n**n 的结果转换为字符串对于大型 n 比实际计算它要耗时得多）。

from multiprocessing import Pool, cpu_count
from timeit import default_timer as timer
import sys


def f(n):
    return n ** n


def main():
    cpu_cnt = cpu_count()
    n = int(sys.argv[2])
    start = timer()
    if sys.argv[1] == "s":
        s = [f(100000) for _ in range(n)]
    else:
        pool = Pool(processes = cpu_cnt)
        s = [pool.map_async(f, (100000,)) for _ in range(n)]
        results = [x.get() for x in s]
    end = timer()
    print(f'time frame in which the operation is resolved: {end - start} seconds')


if __name__ == '__main__':
    main()

以下是我的 4 核机器上 2、6、12、24、48、96 和 192 次函数调用的结果：

% for n in 2 6 12 24 48 96 192; do print $n; for x in s p; do python3 tmp.py $x $n; done; done
2
time frame in which the operation is resolved: 0.146144435 seconds
time frame in which the operation is resolved: 0.178840965 seconds
6
time frame in which the operation is resolved: 0.423103791 seconds
time frame in which the operation is resolved: 0.24940852500000002 seconds
12
time frame in which the operation is resolved: 0.848754817 seconds
time frame in which the operation is resolved: 0.340022419 seconds
24
time frame in which the operation is resolved: 1.691312521 seconds
time frame in which the operation is resolved: 0.571664972 seconds
48
time frame in which the operation is resolved: 3.415401498 seconds
time frame in which the operation is resolved: 1.029526396 seconds
96
time frame in which the operation is resolved: 6.76773454 seconds
time frame in which the operation is resolved: 2.016387216 seconds
192
time frame in which the operation is resolved: 13.529949021999998 seconds
time frame in which the operation is resolved: 3.770171452 seconds

只有 2 个并行进程，由于并行化本身的开销，没有加速。（实际上，速度会变慢。）一旦启动了运行个进程，加速就会增加，但对于 n 个内核，你永远不会看到 [=13] 的加速=].

Multiprocessing Pool() 方法对性能没有影响

Multiprocessing Pool() method has no effect on performance

python

multiprocessing

pool