根据 os.system() 的输出使 build/script 失败

Question

我正在尝试创建 helm charts 并将它们并行（多处理）推送到 100 多个文件夹上的 Nexus 存储库，并且运行良好。

但是，如果退出状态不是 0，我想让脚本或构建失败。使用我当前的代码设置，即使退出代码 returns 非零值，这里是 512，构建最终还是成功的。

预期：即使单个进程失败，构建也会失败..

文件夹结构：

 --/tmp/dir1
      -- values.yaml
      -- zip file to be processed
  --/tmp/dir2
      -- values.yaml
      -- zip file to be processed

   .........
  --/tmp/dirN
      -- values.yaml
      -- zip file to be processed

代码：

#!/usr/bin/env python3

import concurrent.futures
import logging
import sys
import shutil
import os
import glob
import multiprocessing as mp
import traceback
import json
from os import path


def slave(path1, path2, target, logger):
    logging.basicConfig(level=logging.DEBUG, format="%(levelname)s:%(processName)s:%(message)s")

    # create tmp directory where the zip files and yaml files are temporarily copied and follwing commands are ran against them.
    os.makedirs(target)
    logging.info("Create folder %s", target)
    logging.info("Copying files to %s", target)
    try:
        shutil.copy(path1, target)
        shutil.copy(path2, target)
    except:
        traceback.print_exc()

    try:
        result = os.system(<artifactory login command>)
        if result != 0:
            sys.exit(1)
    except:
        traceback.print_exc()

    try:
        # Generate Helm chart and images and push it to Docker registry
        res = os.system("helm create" + target + " --docker-registry dockerio.com/airlines_solutions/ -i")
        if res != 0:
            sys.exit(1)
    except:
        traceback.print_exc()

def main():
    logger = logging.getLogger()
    processed = {}
    with open('example.json', 'r') as f:
        data = json.load(f)
        for value in data.items():
            zip = ""
            yaml = ""
            for line in value[1]:
                if line.endswith('.zip'):
                    zip = line
                elif line.endswith('.yaml'):
                    yaml = line
            processed[zip] = yaml

    with concurrent.futures.ProcessPoolExecutor() as executor:
        for id, (path2, path1) in enumerate(processed.items(), 1):
            target = path.join("/tmp", "dir" + str(id))
            executor.submit(slave, path1, path2, target, logger)
        # waits for the parallel processes to complete & exits at the end.
        executor.shutdown()


if __name__ == "__main__":
    mp.set_start_method('spawn')
    main()

输出：

Error: Invalid value for "manifest": Error: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug="". Either add it there or provide it as an option to this command
Helm chart generation failed
exit code: 512
Traceback (most recent call last):
  File "/home/myud/workspace/jenkins/bin/helm.py", line 65, in worker
    sys.exit(1)
SystemExit: 1

ps -ef |grep python

myud    8483  8454  0 03:36 ?        00:00:00 /home/myud/workspace/venv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=37) --multiprocessing-fork
myud    8484  8454  0 03:36 ?        00:00:00 /home/myud/workspace/venv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=38) --multiprocessing-fork
myud    8485  8454  0 03:36 ?        00:00:00 /home/myud/workspace/venv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=39) --multiprocessing-fork
myud    8486  8454  0 03:36 ?        00:00:00 /home/myud/workspace/venv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=40) --multiprocessing-fork
myud    8487  8454  0 03:36 ?        00:00:00 /home/myud/workspace/venv/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=41) --multiprocessing-fork
myud   10345 10338  0 03:41 ?        00:00:00 grep python

编辑：尝试了另一个选项来捕获 executor.submit 的结果，但是在第一个进程失败后，它退出脚本，而不是等待所有线程或进程完成然后失败构建。

Chart creation failed: <Future at 0x7f2460b40518 state=running>
INFO:Process-2:Create folder /tmp/dir1
INFO:Process-2:Copying files to /tmp/dir1
Error: Invalid value for "manifest": Invalid values /tmp/dir1/values.yaml , version mismatch

替代方法

with concurrent.futures.ProcessPoolExecutor() as executor:
    for id, (path2, path1) in enumerate(processed.items(), 1):
        target = path.join("/tmp", "dir" + str(id))
        result = executor.submit(worker, path1, path2, target, logger)
        if result != 0:
            sys.exit(1)
    # waits for the parallel processes to complete & exits at the end.
    executor.shutdown()

Answer 1

您应该将 subprocess.run 与关键字参数 check=True 一起使用。来自 doc:

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

您的命令将变为：

process = subprocess.run(<command as a list>, check=True, capture_output=True)
result = process.stdout.decode("utf-8")

Answer 2

您应该检查 executor.submit 的结果，如果出现错误，请停止执行程序。

根据 os.system() 的输出使 build/script 失败

Fail the build/script as per os.system()'s output

python

multiprocessing

python-3.x