使用 Joblib 进行多处理：并行化函数的一个参数

Question

有没有一种方法可以让 parallel 函数接受多个参数，但只能并行处理其中一个参数？

假设我有一些代码：

def my_function(graph,graph_coefficients, thing_i_want_to_parallelise_over):

     <do_thing>

     return value 


results = Parallel(n_job2=2)(delayed(my_function(one_graph,graph_coefficients)(thing_i_want_to_parallelise_over) for thing_i_want_to_parallelise_over in range(1,3))

有办法吗？有多个函数可以调用，所以做一个简单的环绕函数并不是一个真正的选择。

Answer 1

我不知道我是否理解你的问题，但你的格式不正确

你应该创建包含所有参数的元组

(one_graph, graph_coefficients, x) for x in range(1,3)   # args

然后你应该将它与

一起使用

delayed( my_function )

喜欢

results = Parallel(n_jobs=2)( 
                delayed( my_function )
                (one_graph, graph_coefficients, x) for x in range(1,3)
          )

最后你可以试试 lambda

lambda x: my_function(one_graph, graph_coefficients,x)

然后你可以使用

(x) for x in range(1,3)

喜欢

results = Parallel(n_jobs=2)(
                delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
                (x) for x in range(1,3) 
          )

或 functools.partial

partial(my_function, one_graph, graph_coefficients)

喜欢

from functools import partial

results = Parallel(n_jobs=2)(
                delayed( partial(my_function, one_graph, graph_coefficients) ) 
                (x) for x in range(1,3) 
          )

最少的工作代码

from joblib import Parallel, delayed

def my_function(graph, graph_coefficients, thing_i_want_to_parallelise_over):
    print('my_function:', graph, graph_coefficients, thing_i_want_to_parallelise_over)
    value = 2 * thing_i_want_to_parallelise_over
    return value 

one_graph = 'A'
graph_coefficients = 'B'

# ----

results = Parallel(n_jobs=2)(
                delayed( my_function ) 
                (one_graph, graph_coefficients, x) for x in range(1,3) 
          )

print('results:', results)

# ----

results = Parallel(n_jobs=2)(
                delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
                (x) for x in range(1,3) 
          )

print('results:', results)

# ----

from functools import partial

results = Parallel(n_jobs=2)(
                delayed( partial(my_function, one_graph, graph_coefficients) ) 
                (x) for x in range(1,3) 
          )

print('results:', results)

使用 Joblib 进行多处理：并行化函数的一个参数

Multiprocessing with Joblib: Parallelising over one argument of a function

python

multiprocessing

joblib