python multiprocessing 如何将父进程全局变量复制到子进程
How are parent process global variables copied to sub-processes in python multiprocessing
Ubuntu 20.04
我对python中不同子进程访问全局变量的理解是这样的：
- 全局变量（比方说
b）在写时复制容量中可供每个子进程使用
- 如果子进程修改了该变量，则首先创建
b 的副本，然后修改该副本。这个变化对父进程是不可见的（稍后我会问这部分的问题）
我做了一些实验试图了解对象何时被复制。我无法得出太多结论：
实验：
import numpy as np
import multiprocessing as mp
import psutil
b=np.arange(200000000).reshape(-1,100).astype(np.float64)
然后我尝试使用下面提到的函数查看内存消耗如何变化：
def f2():
print(psutil.virtual_memory().used/(1024*1024*1024))
global b
print(psutil.virtual_memory().used/(1024*1024*1024))
b = b + 1 ### I changed this statement to study the different memory behaviors. I am posting the results for different statements in place of b = b + 1.
print(psutil.virtual_memory().used/(1024*1024*1024))
p2 = mp.Process(target=f2)
p2.start()
p2.join()
结果格式：
statement used in place of b = b + 1
print 1
print 2
print 3
Comments and questions
结果：
b = b+1
6.571144104003906
6.57244873046875
8.082862854003906
Only a copy-on-write view was provided so no memory consumption till it hit b = b+1. At which point a copy of b was created and hence the memory usage spike
b[:, 1] = b[:, 1] + 1
6.6118621826171875
6.613414764404297
8.108139038085938
Only a copy-on-write view was provided so no memory consumption till it hit b[:, 1] = b[:, 1] + 1. It seems that even if some part of the memory is to be updated (here just one column) the entire object would be copied. Seems fair (so far)
b[0, :] = b[0, :] + 1
6.580562591552734
6.581851959228516
6.582511901855469
NO MEMORY CHANGE! When I tried to modify a column it copied the entire b. But when I try to modify a row, it does not create a copy? Can you please explain what happened here?
b[0:100000, :] = b[0:100000, :] + 1
6.572498321533203
6.5740814208984375
6.656215667724609
Slight memory spike. Assuming a partial copy since I modified just the first 1/20th of the rows. But that would mean that while modifying a column as well some partial copy should have been created, unlike the full copy that we saw in case 2 above. No? Can you please explain what happened here as well?
b[0:500000, :] = b[0:500000, :] + 1
6.593017578125
6.594577789306641
6.970676422119141
The assumption of partial copy was right I think. A moderate memory spike to reflect the change in 1/4th of the total rows
b[0:1000000, :] = b[0:1000000, :] + 1
6.570674896240234
6.5723876953125
7.318485260009766
In-line with partial copy hypothesis
b[0:2000000, :] = b[0:2000000, :] + 1
6.594249725341797
6.596080780029297
8.087333679199219
A full copy since now we are modifying the entire array. This is equal to b = b + 1 only. Just that we have now referred using a slice of all the rows
b[0:2000000, 1] = b[0:2000000, 1] + 1
6.564876556396484
6.566963195800781
8.069766998291016
Again full copy. It seems in the case of row slices a partial copy is getting created and in the case of a column slice, a full copy is getting created which, is weird to me. Can you please help me understand what the exact copy semantics of global variables of a child process are?
如您所见，我没有找到一种方法来证明我在我描述的实验设置中看到的结果。你能帮我理解父进程的全局变量是如何在子进程full/partial修改后复制的吗？
我也：
The child gets a copy-on-write view of the parent memory space. As long as you load the dataset before firing the processes and you don't pass a reference to that memory space in the multiprocessing call (that is, workers should use the global variable directly), then there is no copy.
问题 1：“只要您在启动进程之前加载数据集并且不在多处理调用中传递对该内存 space 的引用（即, workers应该直接使用全局变量)，那么就没有copy了”是什么意思？
下面蒂姆·罗伯茨先生的回答，意思是-
If you pass the dataset as a parameter, then Python has to make a copy to transfer it over. The parameter passing mechanism doesn't use copy-on-write, partly because the reference counting stuff would be confused. When you create it as a global before things start, there's a solid reference, so the multiprocessing code can make copy-on-write happen.
但是，我无法验证此行为。这是我运行验证
的几个测试
import numpy as np
import multiprocessing as mp
import psutil
b=np.arange(200000000).reshape(-1,100).astype(np.float64)
然后我尝试使用下面提到的函数查看内存消耗如何变化：
def f2(b): ### Please notice that the array is passed as an argument and not picked as the global variable of parent process
print(psutil.virtual_memory().used/(1024*1024*1024))
b = b + 1 ### I changed this statement to study the different memory behaviors. I am posting the results for different statements in place of b = b + 1.
print(psutil.virtual_memory().used/(1024*1024*1024))
print(psutil.virtual_memory().used/(1024*1024*1024))
p2 = mp.Process(target=f2,args=(b,)) ### Please notice that the array is passed as an argument and not picked as the global variable of parent process
p2.start()
p2.join()
结果格式：同上
结果：
b = b+1
6.692680358886719
6.69635009765625
8.189273834228516
The second print is arising from within the function hence, by then the copy should have been made and we should see the second print to be around 8.18
b = b
6.699306488037109
6.701808929443359
6.702671051025391
The second and third print should have been around 8.18. The results suggest that no copy is created even though the array b is passed to the function as an argument
写时复制一次做一个虚拟内存页。只要您的更改在一个 4096 字节的页面内，您就只需为该页面付费。当您修改一列时，您的更改会分布在很多很多页面上。我们 Python 程序员不习惯担心物理内存中的布局，但这就是这里的问题。
问题 1：如果您将数据集作为参数传递，那么 Python 必须制作一个副本才能将其传输过来。参数传递机制不使用写时复制，部分原因是引用计数的东西会混淆。当您在事情开始之前将其创建为全局时，就会有可靠的参考，因此多处理代码可以实现写时复制。
Ubuntu 20.04
我对python中不同子进程访问全局变量的理解是这样的：
- 全局变量（比方说
b）在写时复制容量中可供每个子进程使用 - 如果子进程修改了该变量，则首先创建
b的副本，然后修改该副本。这个变化对父进程是不可见的（稍后我会问这部分的问题）
我做了一些实验试图了解对象何时被复制。我无法得出太多结论：
实验：
import numpy as np
import multiprocessing as mp
import psutil
b=np.arange(200000000).reshape(-1,100).astype(np.float64)
然后我尝试使用下面提到的函数查看内存消耗如何变化：
def f2():
print(psutil.virtual_memory().used/(1024*1024*1024))
global b
print(psutil.virtual_memory().used/(1024*1024*1024))
b = b + 1 ### I changed this statement to study the different memory behaviors. I am posting the results for different statements in place of b = b + 1.
print(psutil.virtual_memory().used/(1024*1024*1024))
p2 = mp.Process(target=f2)
p2.start()
p2.join()
结果格式：
statement used in place of b = b + 1
print 1
print 2
print 3
Comments and questions
结果：
b = b+1
6.571144104003906
6.57244873046875
8.082862854003906
Only a copy-on-write view was provided so no memory consumption till it hit b = b+1. At which point a copy of b was created and hence the memory usage spike
b[:, 1] = b[:, 1] + 1
6.6118621826171875
6.613414764404297
8.108139038085938
Only a copy-on-write view was provided so no memory consumption till it hit b[:, 1] = b[:, 1] + 1. It seems that even if some part of the memory is to be updated (here just one column) the entire object would be copied. Seems fair (so far)
b[0, :] = b[0, :] + 1
6.580562591552734
6.581851959228516
6.582511901855469
NO MEMORY CHANGE! When I tried to modify a column it copied the entire b. But when I try to modify a row, it does not create a copy? Can you please explain what happened here?
b[0:100000, :] = b[0:100000, :] + 1
6.572498321533203
6.5740814208984375
6.656215667724609
Slight memory spike. Assuming a partial copy since I modified just the first 1/20th of the rows. But that would mean that while modifying a column as well some partial copy should have been created, unlike the full copy that we saw in case 2 above. No? Can you please explain what happened here as well?
b[0:500000, :] = b[0:500000, :] + 1
6.593017578125
6.594577789306641
6.970676422119141
The assumption of partial copy was right I think. A moderate memory spike to reflect the change in 1/4th of the total rows
b[0:1000000, :] = b[0:1000000, :] + 1
6.570674896240234
6.5723876953125
7.318485260009766
In-line with partial copy hypothesis
b[0:2000000, :] = b[0:2000000, :] + 1
6.594249725341797
6.596080780029297
8.087333679199219
A full copy since now we are modifying the entire array. This is equal to b = b + 1 only. Just that we have now referred using a slice of all the rows
b[0:2000000, 1] = b[0:2000000, 1] + 1
6.564876556396484
6.566963195800781
8.069766998291016
Again full copy. It seems in the case of row slices a partial copy is getting created and in the case of a column slice, a full copy is getting created which, is weird to me. Can you please help me understand what the exact copy semantics of global variables of a child process are?
如您所见，我没有找到一种方法来证明我在我描述的实验设置中看到的结果。你能帮我理解父进程的全局变量是如何在子进程full/partial修改后复制的吗？
我也
The child gets a copy-on-write view of the parent memory space. As long as you load the dataset before firing the processes and you don't pass a reference to that memory space in the multiprocessing call (that is, workers should use the global variable directly), then there is no copy.
问题 1：“只要您在启动进程之前加载数据集并且不在多处理调用中传递对该内存 space 的引用（即, workers应该直接使用全局变量)，那么就没有copy了”是什么意思？
下面蒂姆·罗伯茨先生的回答，意思是-
If you pass the dataset as a parameter, then Python has to make a copy to transfer it over. The parameter passing mechanism doesn't use copy-on-write, partly because the reference counting stuff would be confused. When you create it as a global before things start, there's a solid reference, so the multiprocessing code can make copy-on-write happen.
但是，我无法验证此行为。这是我运行验证
的几个测试import numpy as np
import multiprocessing as mp
import psutil
b=np.arange(200000000).reshape(-1,100).astype(np.float64)
然后我尝试使用下面提到的函数查看内存消耗如何变化：
def f2(b): ### Please notice that the array is passed as an argument and not picked as the global variable of parent process
print(psutil.virtual_memory().used/(1024*1024*1024))
b = b + 1 ### I changed this statement to study the different memory behaviors. I am posting the results for different statements in place of b = b + 1.
print(psutil.virtual_memory().used/(1024*1024*1024))
print(psutil.virtual_memory().used/(1024*1024*1024))
p2 = mp.Process(target=f2,args=(b,)) ### Please notice that the array is passed as an argument and not picked as the global variable of parent process
p2.start()
p2.join()
结果格式：同上
结果：
b = b+1
6.692680358886719
6.69635009765625
8.189273834228516
The second print is arising from within the function hence, by then the copy should have been made and we should see the second print to be around 8.18
b = b
6.699306488037109
6.701808929443359
6.702671051025391
The second and third print should have been around 8.18. The results suggest that no copy is created even though the array b is passed to the function as an argument
写时复制一次做一个虚拟内存页。只要您的更改在一个 4096 字节的页面内，您就只需为该页面付费。当您修改一列时，您的更改会分布在很多很多页面上。我们 Python 程序员不习惯担心物理内存中的布局，但这就是这里的问题。
问题 1：如果您将数据集作为参数传递，那么 Python 必须制作一个副本才能将其传输过来。参数传递机制不使用写时复制，部分原因是引用计数的东西会混淆。当您在事情开始之前将其创建为全局时，就会有可靠的参考，因此多处理代码可以实现写时复制。