如何在 python 中并行化子类中的进程

Question

我正在做一个项目，其中有一个主要功能，一个名为 simulator 的 class 和一个名为 vehicle 的 class。在我调用的主要函数中：simulator.run() 为 simulator.vehicle_list[].

中的所有车辆调用 vehicle.run()

车辆必须计算轨迹，这是一个耗时的过程。这些进程彼此独立，我希望 vehicle.run() 计算在多个 CPU 核心上进行运行。

作为初学者，我创建了一个简单的项目（使用与上面不同的 class-名称，但我希望足够直观）但我无法让它工作。我的代码没有在 subclass 中调用 self.calculate()。当我从 master class 中删除 if __name__ == '__main__': 语句时，出现错误：

'multi_subclass' object has no attribute '_closed'

我拥有的三个 python 文件：

multi_main.py

from multi_class import multi_class

main_class = multi_class()
main_class.calculate()

#main_class.calculate2()

multi_class.py

from multi_subclass import multi_subclass

class multi_class():
    def __init__(self):
        print("I am multi_class")
        subclass_list = []
        for i in range(10):
            subclass_list += [multi_subclass()]
        self.subclass_list = subclass_list
        
    def calculate(self):
        if __name__ == '__main__':
            processes = []
            for i in range(10):
                processes.append(self.subclass_list[i])
                self.subclass_list[i].start()
                
            [proc.join() for proc in processes]

multi_subclass.py

from multiprocessing import Process

class multi_subclass(Process):
    def __init__(self):
        print("I am multi_subclass")
        
    def calculate(self):
        print("Running massive calculations")
        self.member_variable = "Finished with calculation"
        
    def calculate2(self):
        print("I also want to be paralellized but NOT together with calculate!")
        
    def run(self):
        self.calculate()

Answer 1

好的，所以这条语句 if __name__ == '__main__': 就位，这样如果主进程产生新进程，这些新进程也不会产生新进程。

所以，您的入口点应该有那个。

multi_main.py

from multi_class import multi_class
from timeit import default_timer


start = default_timer()

if __name__ == '__main__':

    main_class = multi_class()
    main_class.calculate()
    print(f"Your code ran in {default_timer() - start} seconds instead of 10 seconds because of multiprocessing.LOL")

multi_class.py <-- Class 负责启动新流程。 calculate() 函数中存在一些问题。

from multi_subclass import multi_subclass

class multi_class():
    def __init__(self):
        print("I am multi_class")
        subclass_list = []
        for i in range(10):
            subclass_list += [multi_subclass()]
        self.subclass_list = subclass_list

    def calculate(self):
        processes = []

        for i in range(10):
            # you were putting a different process in the list and never starting it but you were waiting for it to complete using `.join()` and starting a different process but never waiting for it to complete
            # fixed code below
            # processes.append(self.subclass_list[i])
            # self.subclass_list[i].start() 
            

            # create a process, put it in the list, start it
            process = self.subclass_list[i]
            processes.append(process)
            process.start()

        # wait for it to complete here
        [proc.join() for proc in processes]

multi_subclass.py < -- 最重要的 class 我喜欢你继承流程的想法 class，所以现在你的 class 也是一个进程 class。酷

但问题是，当您使用 multiprocessing 模块创建 Process() 时，当进程像这样启动时，您向运行提供了一个目标 -> Process(target=abc) abc 是一个函数。

但是在您的代码中，__init__ 不会调用 Process class 的 __init__ 并且返回的对象是一个没有目标的简单对象。我已经添加了代码。

from multiprocessing import Process
import time
class multi_subclass(Process):
    def __init__(self):
        print("I am multi_subclass")
        super().__init__(target=self.calculate)

    def calculate(self):
        print("Running massive calculations")
        time.sleep(1)
        self.member_variable = "Finished with calculation"


    def calculate2(self):
        print("I also want to be paralellized but NOT together with calculate!")

    def run(self):
        self.calculate()

输出

I am multi_class
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
I am multi_subclass
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Running massive calculations
Your code ran in 1.0859881550000001 seconds instead of 10 seconds because of multiprocessing.LOL

如何在 python 中并行化子类中的进程

How to parallelize a process in a subclass in python

python

multiprocess

python-multithreading