在 Python 中调用多个 linux 进程并收集输出

Question

从 Python 脚本，我需要调用 PL->EN 翻译服务。翻译需要 3 个步骤：tokenization、translation、detoknization

从 Linux 开始，我可以通过按上述顺序执行以下命令使用 3 个进程来实现此目的：

/home/nlp/opt/moses/scripts/tokenizer/tokenizer.perl -l pl < path_to_input.txt > path_to_output.tok.txt

/home/nlp/opt/moses/bin/moses -f /home/nlp/Downloads/TED/tuning/moses.tuned.ini.1 -drop-unknown -input-file path_to_output.tok.txt -th 8 > path_to_output.trans.txt

/home/nlp/opt/moses/scripts/tokenizer/detokenizer.perl -l en < path_to_output.trans.txt > path_to_output.final.txt

翻译文件 path_to_input.txt 并输出到 path_to_output.final.txt

我制作了以下脚本来组合这 3 个进程：

import shlex
import subprocess
from subprocess import STDOUT,PIPE
import os
import socket

class Translator:
    @staticmethod
    def pl_to_en(input_file, output_file):
        # Tokenize
        print("Tokenization started")
        with open("tokenized.txt", "w+") as tokenizer_output:
            with open(input_file) as tokenizer_input:
                cmd = "/home/nlp/opt/moses/scripts/tokenizer/tokenizer.perl -    l pl"
                args = shlex.split(cmd)
                p = subprocess.Popen(args, stdin=tokenizer_input, stdout=tokenizer_output)
                p.wait()
                print("Tokenization finished")

        #Translate
        print("Translation started")
        with open("translated.txt", "w+") as translator_output:
            cmd = "/home/nlp/opt/moses/bin/moses -f /home/nlp/Downloads/TED/tuning/moses.tuned.ini.1 -drop-unknown -input-file tokenized.txt -th 8"
            args = shlex.split(cmd)
            p = subprocess.Popen(args, stdout=translator_output)
            p.wait()
            print("Translation finished")

        # Detokenize
        print("Detokenization started")
        with open("translated.txt") as detokenizer_input:
            with open("detokenized.txt", "w+") as detokenizer_output:
                cmd = "/home/nlp/opt/moses/scripts/tokenizer/detokenizer.perl -l en"
                args = shlex.split(cmd)
                p = subprocess.Popen(args, stdin=detokenizer_input, stdout=detokenizer_output)
                p.wait()
                print("Detokenization finished")

translator = Translator()
translator.pl_to_en("some_input_file.txt", "some_output_file.txt")

但只有标记化部分有效。翻译器只是输出一个空文件translated.txt。在终端中查看输出时，翻译器似乎正确加载了文件 tokenized.txt，并进行了翻译。问题在于我如何收集该过程的输出。

Answer 1

我会尝试类似下面的操作 - 将翻译器进程的输出发送到管道，并使去标记器的输入成为管道而不是使用文件。

import shlex
import subprocess
from subprocess import STDOUT,PIPE
import os
import socket

class Translator:
    @staticmethod
    def pl_to_en(input_file, output_file):
        # Tokenize
        print("Tokenization started")
        with open("tokenized.txt", "w+") as tokenizer_output:
            with open(input_file) as tokenizer_input:
                cmd = "/home/nlp/opt/moses/scripts/tokenizer/tokenizer.perl -    l pl"
                args = shlex.split(cmd)
                p = subprocess.Popen(args, stdin=tokenizer_input, stdout=tokenizer_output)
                p.wait()
                print("Tokenization finished")

        #Translate
        print("Translation started")
        cmd = "/home/nlp/opt/moses/bin/moses -f /home/nlp/Downloads/TED/tuning/moses.tuned.ini.1 -drop-unknown -input-file tokenized.txt -th 8"
        args = shlex.split(cmd)
        translate_p = subprocess.Popen(args, stdout=subprocess.PIPE)
        translate_p.wait()
        print("Translation finished")
    # Detokenize
        print("Detokenization started")
        with open("detokenized.txt", "w+") as detokenizer_output:
            cmd = "/home/nlp/opt/moses/scripts/tokenizer/detokenizer.perl -l en"
            args = shlex.split(cmd)
            detokenizer_p = subprocess.Popen(args, stdin=translate_p.stdout, stdout=detokenizer_output)
            detokenizer_p.wait()
            print("Detokenization finished")

translator = Translator()
translator.pl_to_en("some_input_file.txt", "some_output_file.txt")

在 Python 中调用多个 linux 进程并收集输出

Calling multiple linux processes in Python and collecting output

python

external

process

output