IPython ipyparallel map_sync 导入错误

Question

我是 ipyparallel 的新手，我想使用这个包来实现我的机器学习应用程序的并行计算。

下面是在ipyparallel上的测试，我在func.py文件上定义了一个名为add的函数，在test.py文件上定义了main函数[=26] =]

func.py的代码是：

#!/usr/bin/env python
# coding=utf-8

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers

test.py的代码是：

#!/usr/bin/env python
# coding=utf-8

from func import add
from ipyparallel import Client

if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3]
    print rc[0].map_sync(add, [1, 2, 3, 4])

既然知道map可以运行不报错，但是运行map_sync时，命令行returns:

☁  test  python test.py 
[[2], [3], [4]]
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print rc[0].map_sync(add, [1, 2, 3, 4])
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 353, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 54, in sync_results
    ret = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 618, in map
    return pf.map(*sequences)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 268, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 75, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 251, in __call__
    return r.get()
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/asyncresult.py", line 104, in get
    raise self._exception
ipyparallel.error.CompositeError: one or more exceptions from call to method: add
[0:apply]: ImportError: No module named func

如果我在 test.py 文件上定义函数，map_sync 可以运行:

#!/usr/bin/env python
# coding=utf-8

#from func import add
from ipyparallel import Client

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers


if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3])

    print rc[0].map_sync(add, [1, 2, 3, 4])

结果是：

☁  test  python test.py
[[2], [3], [4]]
[[2], [3], [4], [5]]

我想知道map_sync如何使用在其他文件中定义的函数？以及如何导入这些功能？由于 from py_file import func 不适用于 map_sync。

Answer 1

应将所需的模块复制（或者，您可以推送或模块）到 engine machines，并且应在 engine machines 上安装 3 方包，如果没有，ImportError将会发生。

但是，当运行运行程序时，您应该运行:

$ ipcontroller --ip=client_ip --reuse

在client machine，/home/user/.ipython/profile_default/security目录下会生成2个文件

$ ls /home/user/.ipython/profile_default/security 
ipcontroller-client.json  ipcontroller-engine.json

因此，ipcontroller-client.json 和 ipcontroller-engine.json 都必须复制到 engine machines 和运行

$ ipengine --file=/path/to/ipcontroller-engine.json

于engine machines，因此parallel computing环境已设置。

接下来，您可以定义 parallel computing 任务和运行程序。

IPython ipyparallel map_sync 导入错误

IPython ipyparallel map_sync ImportError

python

ipython-parallel