Scipy.Optimize - 多重处理并在 Python 中的最后一次迭代中恢复

Question

Scipy.Optimize 可以选择通过将 "workers" 参数更改为 > 1（或 -1）来启用多处理。但是，如果我希望能够从上一次迭代中恢复优化，同时仍然使用多处理功能，这似乎是不可能的。

我可以通过使用 DifferentialEvolutionSolver 恢复操作（根据 https://github.com/scipy/scipy/issues/6517），但是在 DifferentialEvolutionSolver 中 "workers" > 1 使我无法 pickle 对象以保持会话之间的持久性。

import pickle
from scipy.optimize._differentialevolution import DifferentialEvolutionSolver  

bounds = [(-1, 1)]
bounds = bounds * 66

if __name__ == '__main__':

    solver = DifferentialEvolutionSolver(converter, bounds, disp=True, seed=9, workers=-1, 
    maxiter=1)
    for i in range(100):
            best_x, best_cost = next(solver)
            print(solver.population_energies.min())
            with open('solver_%d.pkl' % i, 'wb') as f:
                 pickle.dump(solver, f)

此代码在尝试 pickle 其第一个运行时会生成以下错误：

NotImplementedError: pool objects cannot be passed between processes or pickled

但是，如果我使用 "workers=1"，代码工作正常，但显然要慢得多。

有没有办法同时获得多处理和保存每次迭代的能力？

Answer 1

请记住，DifferentialEvolutionSolver 不是 SciPy 的 public API 的一部分，它可能会更改.提高性能需要更改的能力，或 re-engineering。向后兼容的 public 面向函数是 differential_evolution。如果您准备好应对这种情况，那么您可以使用以下方法：

import pickle
from multiprocessing import Pool

from scipy.optimize import rosen
from scipy.optimize._differentialevolution import DifferentialEvolutionSolver  

bounds = [(-2, 2), (-2, 2)]

with Pool(2) as p:
    # make sure that `updating='deferred'`, as this is required for
    # parallelisation
    solver = DifferentialEvolutionSolver(
        rosen,
        bounds,
        seed=9,
        workers=1,
        updating='deferred',
    )

    for i in range(100):
        # the _mapwrapper attribute needs to be a map-like
        solver._mapwrapper = p.map
        best_x, best_cost = next(solver)
        print(solver.population_energies.min())
        solver._mapwrapper = map
        with open(f"solver_{i}.pkl", 'wb') as f:
             pickle.dump(solver, f)

Scipy.Optimize - 多重处理并在 Python 中的最后一次迭代中恢复

Scipy.Optimize - multiprocessing and resuming on last iteration in Python

python

pickle

python-multiprocessing

scipy-optimize