gnu parallel：在 sigint 上打印部分输出

Question

我使用 gnu 并行运行测试开发中程序的输入。简化设置：

find . -iname '*test*' | parallel -k -I@ echo running '@' \; my_binary \; echo '@': exited with $?

如果my_binary陷入无限循环，我Ctrl+C -> SigINT这个测试脚本。

在这里，不幸的是，并行吃掉了所有（已打印，但已缓冲以供订购）stdout，所以我看不出哪个测试卡住了。

我想获取此标准输出，但我在手册页中找不到执行该操作的选项。您对如何执行此操作有想法吗？

我想保持输出的顺序（这里是由-k引起的），所以仅仅将所有输出直接重定向到一个文件是行不通的（除非有一个我不知道的技巧？）。
为每个测试设置某种超时是不切实际的

（有些测试需要很长时间，但你知道有一个问题很漂亮由于 cpu 使用等原因很快。
我不太关心使用 gnu-parallel，任何解决方案都很好。

Answer 1

好的，我自己想出来了。

parallel 有一个 --joblog 选项可以立即打印所有已完成的作业，但仅按提交顺序！

结合@Mark Setchell 的标签创意，这正是我们想要的对于以下概念证明：


output="$(mktemp)" # parallel will put the line buffered command output here
joblog="$(mktemp)" # parallel will output it's joblog here for sigint recovery

function run_test () {
   tag=""
   input=""
   echo "$tag: started $input"
   # simulate doing work for demonstration
   sleep "$tag"
   # output from the test must have the tag prepended to each line 
   echo "$input: test output"  2>&1 | sed "s/^/$tag: /" 
   echo "$tag: finished $input, exited with $?"
}

function handle_sigint () {
    echo " interrupted!"
    # check for finished jobs (tag in the joblog)
    awk '(NR>1){printf "^%s: \n", }' "$joblog"  | 
        # remove all output lines with tags from finished jobs
        grep -v -f - "$output" | 
        # print the remaining output lines, sorted by tag, but in order of line submission
        sort -snk1 |
        # but strip away the tag from the beginning of the line
        cut -d" " -f2-
    rm "$output"
    rm "$joblog"
    exit 0
}

export -f run_test
trap handle_sigint INT

# create some dummy jobs for demonstration
printf "foo\nbar\nbaz\nquux\n" | 
    # create command to run for each test
    awk '{printf "run_test %i %s\n", NR, }' | 
    # run the jobs, putting line buffered output into $output and the joblog into $joblog
    parallel --line-buffered --joblog "$joblog" bash -c  &>"$output" & 
# immedeately pipe new joblog lines along until parallel dies or we sigint
tail -f "$joblog" --pid $! |
    # whenever a job is finished --> a joblog line is added, print all lines with that tag from the output
    awk "(NR>1){c=sprintf(\"sed -n 's/^%s: //p' '%s'\n\" , $1, \"$output\"); system(c)}" 

# remove the temp files (if we were not interrupted, otherwise handle_sigint does this)
rm "$output" 
rm "$joblog"

Answer 2

询问 GNU Parallel 当前有哪些作业如何运行:

seq 1000 | parallel sleep &
pid=$!
sleep 10
kill -SIGUSR1 $pid

gnu parallel：在 sigint 上打印部分输出

gnu parallel: print partial output on sigint

linux

bash

shell

gnu-parallel