如何在多达 3400 台机器上快速执行脚本？

Question

我有 3400 台机器需要向 Graphite 发送数据，但是我的脚本需要大约 5 分钟才能完成，我怎样才能将它减少到 3 或 2 分钟，谢谢，代码如下

def sendmsg(hostname,metric):
        a = mntdir + hostname + '/' + metric + '_median'
        with settings(hide('running'), warn_only=True):
            valueMedian = local(("cat %s|grep %s|awk '{print }'") % (a,now),capture=True)
            local(("echo %s.%s_Median_90days %s %s >/dev/udp/20.26.2.18/2001 ") % (hostname,metric,valueMedian,unixdate))


if __name__=='__main__':
    while True:
        localtime = time.asctime( time.localtime(time.time()) )
        date = datetime.datetime.now()
        now1 = date.strftime("%H:%M")
        print now1
        now2 = date.strftime("%M")
        with settings(hide('running'), warn_only=True):
            unixdate = local("date +%s",capture=True)
        if int(now2) % 5 == 0:
            now = now1 + ':00'
            p=Pool(100)
            print now
            for hostname in host:
                for metric in metrics:
                    p.apply_async(sendmsg, args=(hostname,metric))
            starttime = time.asctime( time.localtime(time.time()) )
            print('Waiting for all job done...%s' % starttime)
            p.close()
            p.join()
            stoptime = time.asctime( time.localtime(time.time()) )
            print('sending completed...%s ' % stoptime)
        time.sleep(60)

Answer 1

awk '/foo/ {print }' file

与您的

大致相同

cat file | grep foo | awk '{print }'

并避免创建两个进程，这可能会有所帮助。

如何在多达 3400 台机器上快速执行脚本？

How to fast the scripts with up to 3400 machines?

python

multiprocessing

fabric