Python 从另一个 python 脚本调用 python 脚本时日志记录挂起

Question

我在 python 日志 class 中观察到这个奇怪的问题，我有两个脚本，一个是从另一个调用的。第一个脚本等待其他脚本结束，而其他脚本使用 logging.info

记录大量日志

这是代码片段

#!/usr/bin/env python

import subprocess
import time
import sys

chars = ["/","-","\","|"]
i = 0
command = 'sudo python /home/tejto/test/writeIssue.py'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
while process.poll() is None:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
output = process.communicate()

另一个脚本是

#!/usr/bin/env python

import os
import logging as log_status

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
            log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
    self.upgradethreadstatus.status_change("Entered upgrade routine")
    procoutput = 'very huge logs, mine were 145091 characters'
    self.upgradethreadstatus.status_change(procoutput)
    self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:]

如果我运行第一个脚本，两个脚本都会挂起，问题似乎出在代码 while process.poll() is None 上，如果我注释这段代码，一切正常。（无法将此与我的问题联系起来！！）

PS 我也尝试调试 python 日志记录 classes，我发现进程卡住了 emit 函数StreamHandlerclass，其中它卡在 stream.write 函数调用并且在写入大量日志后没有出现，但是我的退出日志没有出现。

那么导致死锁情况的这些脚本可能是什么问题？

Edit 1 (code with threading)

script.py

#!/usr/bin/env python 

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
    output = process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\","|"]
i = 0

while t.is_alive:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
t.join()

script2.py

#!/usr/bin/env python  
import os
import sys
import logging as log_status
import time

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                        os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
        log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        procoutput = "Please put logs of 145091 characters over here otherwise the situtation wouldn't remain same or run any command whose output is larger then 145091 characters"
    self.upgradethreadstatus.status_change(procoutput)
        time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:])

在这种情况下，t.is_alive 函数不是 returning false（不知道，但是启动函数已经 returned 所以理想情况下它应该 return false!! :()

Answer 1

stdout 缓冲区便秘。

process.communicate() 调用未执行，因为它在之后 while process.poll() is None:。所以 writeIssue.py 试图向 stdout 写入太多字节，并且所有字节都在 subprocess.PIPE 中缓冲，并且在调用 communicate 之前不会从 PIPE 中拉出。

缓冲区大小有限。当缓冲区满时，stream.write会阻塞直到缓冲区有 space。如果缓冲区永远不会被清空（就像发生在你的代码），然后进程死锁。

修复方法是在缓冲区完全填满之前调用 communicate()。您可以通过在线程中启动 writeIssue.py 并同时调用 communicate() 而 while-thread-is-alive 循环在主线程中运行线程。

script.py:

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command)
    process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\","|"]
i = 0
while t.is_alive():
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4

t.join()

script2.py:

import sys
import logging
import time

class UpgradeStatus():
    def __init__(self):
        logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.DEBUG)

        consoleHandler = logging.StreamHandler()
        consoleHandler.setLevel(logging.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        self.logger.addHandler(consoleHandler)

    def status_change(self, status):
        self.logger.info(str(status))

class UpgradeThread():
    def __init__(self, link):
        self.upgradethreadstatus = UpgradeStatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        for i in range(5):
            procoutput = 'very huge logs, mine were 145091 characters'
            self.upgradethreadstatus.status_change(procoutput)
            time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = UpgradeThread(sys.argv[1:])

请注意，如果您有两个线程同时写入 stdout，则输出会乱码。如果你想避免这种情况，那么所有输出都应该由一个 单线程 配备队列。所有其他希望的线程或进程写输出应该将字符串或日志记录推送到队列中专用输出线程来处理。

然后该输出线程可以使用 for 循环从队列中提取输出：

for message from iter(queue.get, None):
    print(message)

Python 从另一个 python 脚本调用 python 脚本时日志记录挂起

Python logging hangs when calling a python script from another python script

python

logging

deadlock

subprocess

python-2.7

script.py

script2.py