ProcessPoolExecutor 锁定了比必要更多的期货

Question

我正在使用 ProcessPoolExecutor 生成子进程。目前我正在尝试通过 KeyboardInterrupt / Ctrl+C 优雅地退出脚本。

我正在用 2 个工人创建池并提交 5 个期货。在中断时，我试图取消尚未执行的所有未来。如果我在前两个期货的执行过程中中断，池子只能取消两个期货，这意味着三个目前是运行。但是我只有两个工人，每个进程运行 5 秒。我的未来在执行什么或为什么执行？

import subprocess
from concurrent.futures import ProcessPoolExecutor
import signal
import sys


def executecommands(commands):
    # Replace signal handler of parent process, so child processes will ignore terminate signals
    original_sigint_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
    processpool = ProcessPoolExecutor(1)
    # Restore original handler, so the parent process can handle terminate signals
    signal.signal(signal.SIGINT, original_sigint_handler)
    futures = []
    try:
        for command in commands:
            futures.append(processpool.submit(executecommand, command))

        processpool.shutdown()
        print("Executed commands without interruption")
    except KeyboardInterrupt:
        print("\nAttempting to cancel pending commands..")
        for future in futures:
            if future.cancel():
                print("Cancelled one command")
            else:
                print("One command was already running, couldn't cancel")
        print("Waiting for running processes to finish..")
        processpool.shutdown()
        print("Shutdown complete")
        sys.exit(0)


def executecommand(command):
    # create a subprocess and run it
    print("running command")
    process = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    print("finished command")
    return process

if __name__ == '__main__':
    print("lets go")
    commandlist = [['/bin/sleep', '5'], ['/bin/sleep', '5'], ['/bin/sleep', '5'], ['/bin/sleep', '5'], ['/bin/sleep', '5']]
    executecommands(commandlist)

Answer 1

这是一个 CPython 实现细节，但您唯一可以取消的期货是那些不在 "call queue" 中的期货。调用队列包含接下来要执行的所有期货。它的大小是max_workers + EXTRA_QUEUED_CALLS。 (EXTRA_QUEUED_CALLS is currently set to 1.)

在你的情况下，当你的前两个期货开始执行时，调用队列中充满了接下来的 3 个期货（max_workers 是 2，EXTRA_QUEUED_CALLS 是 1）。由于您总共只有 5 个期货，因此您不能取消任何一个。

如果您用 2 个工人的 10 个期货填充您的命令列表，您将能够取消最后五个期货：

lets go
running command
running command
^C
Attempting to cancel pending commands..
One command was already running, couldn't cancel
One command was already running, couldn't cancel
One command was already running, couldn't cancel
One command was already running, couldn't cancel
One command was already running, couldn't cancel
Cancelled one command
Cancelled one command
Cancelled one command
Cancelled one command
Cancelled one command
Waiting for running processes to finish..
running command
running command
finished command
finished command
running command
finished command
Shutdown complete

ProcessPoolExecutor 锁定了比必要更多的期货

ProcessPoolExecutor locks more futures than necessary

python

concurrency

subprocess

terminate

python-3.x