使用多处理时如何将 env.host 传输到 fabric？

Question

我有几百个盒子只能通过不同的网关访问，如下所示：

gateway1:ip1,ip2
gateway2:ip3,ip4 ...

所有工作需要在一分钟内完成，所以我在命令 fab -f ytj_sto.py doitnow 下使用多进程，错误如下。

[]
None
None
***Warning*** Host None via ssh is down

代码：

@parallel(pool_size=20)
def coll():
    print env.hosts
    print env.host
    print env.gateway
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = p.apply_async(coll, args=())
            result.get()
    p.close()
    p.join()

已编辑：我已经使用 fab -H -g 解决了这个问题，谢谢大家

  def fabfun(Hosts,Gate,des,func1):
        with settings(hide('running'), warn_only=True):
           local(("fab -H %s -g %s -f %s %s ") % (Hosts,Gate,des,func1))

p=Pool(20)
starttime = time.asctime( time.localtime(time.time()) )
print('Waiting for all job done...%s' % starttime)
with open('ytj_sto.hn','r') as f:
    for line in f.readlines():
        line = line.strip()
        if not len(line) or line.startswith('#'):
            continue
        Hosts = line.split(':')[1]
        Gate = line.split(':')[0]
        p.apply_async(fabfun, args=(Hosts,Gate,des,func1))
    p.close()
    p.join()

Answer 1

我认为您可以将 env 作为参数提供给 coll 函数，如下所示：

@parallel(pool_size=20)
def coll(env): # <-- updated
    print env.hosts
    print env.host
    print env.gateway
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = p.apply_async(coll, args=(env,)) # <-- updated
            result.get()
    p.close()
    p.join()

使用 multiprocessing library 有一些怪癖。此信息可能与您的情况特别相关：

Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.

However, global variables which are just module level constants cause no problems.

Answer 2

如果你想像现在一样动态设置环境变量，你应该使用execute。这样，executed 任务就会采用您在运行时设置的 env 值。但不幸的是，因为 fabric 不是完全线程安全的，你只能在你的情况下为每个网关并行启动一次这样的任务，因为 env 是一个全局单例。

A simple but integral aspect of Fabric is what is known as the “environment”: a Python dictionary subclass, which is used as a combination settings registry and shared inter-task data namespace.

The environment dict is currently implemented as a global singleton, fabric.state.env, and is included in fabric.api for convenience. Keys in env are sometimes referred to as “env variables”.

from fabric.context_managers import env

@parallel(pool_size=20)
def check_and_patt():
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = execute(check_and_patt)

使用多处理时如何将 env.host 传输到 fabric？

How to transfer env.host to fabric when using multiprocessing?

python

fabric