锁定进程直到子进程到达某个点

Question

我正在运行并行设置多个子进程，但我需要锁定每个进程，直到子进程给出输出（通过打印函数）。子进程是运行ning 已打包为可执行文件的 python 脚本。

代码如下所示：

import multiprocessing as mp
import subprocess
import os


def main(args):

    l,inpath = args
    l.acquire()
    print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\')[-1] 

    #Run KNN executable as a subprocess
    subprocess.call(os.path.join(os.getcwd(), "KNN.exe"))
    #This is where I want to wait for any output from the subprocess before releasing the lock
    l.release()

    #Here I would like to wait until subprocess is done then print that it is done
    l.acquire()
    print "Done %s" % os.path.normpath(inpath).split('\')[-1]
    l.release()


if __name__ == "__main__":
    #Set working directory path containing input text file
    os.chdir("C:\Users\Patrick\Google Drive\KNN")
    #Get folder names in directory containing GCM input
    manager = mp.Manager()
    l = manager.Lock()
    gcm_dir = "F:\FIDS_GCM_Data_CMIP5\UTRB\UTRB KNN-CAD\Input"
    paths = [(l, os.path.join(gcm_dir, folder)) for folder in os.listdir(gcm_dir)]    
    #Set up multiprocessing pool
    p = mp.Pool(mp.cpu_count())
    #Map function through input paths
    p.map(main, paths)

所以目标是锁定进程，以便子进程可以运行直到收到输出。之后可以释放锁，子进程可以继续，直到完成，然后我想打印完成。

我的问题是如何在释放对进程的锁定（多个）之前等待子进程的单个（且唯一）输出？

此外，我如何等待进程终止然后打印它已完成？

Answer 1

您的代码使用了 call 方法，该方法已经在等待子进程完成（这意味着所有输出都已生成）。我从你的问题中推断出你希望能够区分何时首次写入输出和何时完成子流程。以下是您的代码，其中包含我建议的内联修改：

def main(args):

    l,inpath = args
    l.acquire()
    print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\')[-1]

    #Run KNN executable as a subprocess
    #Use the Popen constructor
    proc = subprocess.Popen(os.path.join(os.getcwd(), "KNN.exe"), stdout=subprocess.PIPE)

    #This is where I want to wait for any output from the subprocess before releasing the lock
    # Wait until the subprocess has written at least 1 byte to STDOUT (modify if you need different logic)
    proc.stdout.read(1)

    l.release()

    #Here I would like to wait until subprocess is done then print that it is done
    #proc.wait()
    (proc_output, proc_error) = proc.communicate()

    l.acquire()
    print "Done %s" % os.path.normpath(inpath).split('\')[-1]
    l.release()

请注意，除了检查它是否已生成之外，上面的内容并不假设您想要对子进程的输出执行任何操作。如果你想对那个输出做任何比上面更简单的事情（消耗 1 个字节然后把它放在地板上），proc.stdout（这是一个 file 对象）应该代表子进程生成 while 运行.

锁定进程直到子进程到达某个点

Lock process until subprocess reaches a certain point

python

locking

multiprocessing

subprocess