Nextflow，限制 Google 中的并发工作人员数量

Question

我想知道在使用 Google Life Sciences 执行程序时是否有办法限制并发运行工作人员的数量。我在 nextflow.config 文件中使用了以下内容，但显然这无关紧要。

executor {
    queueSize = 100
}

同时有超过100个工人运行（进程名称旁边大括号中的数字显示108）。

Answer 1

像您所做的那样设置executor.queueSize = 100 应该准确地限制执行程序将处理的并行任务的数量。但是，我认为您遇到的问题实际上是由于以下原因造成的：

The channel guarantees that items are delivered in the same order as they have been sent - but - since the process is executed in a parallel manner, there is no guarantee that they are processed in the same order as they are received.

即使 queueSize 设置为 1 也是如此。结果可能出乎意料，但我们可以很容易地测试它：

nextflow.config 的内容：

executor {
    queueSize = 1
}

main.nf 的内容：

nextflow.enable.dsl=2

process test {

    tag { "myval: ${myval}" }

    input:
    val myval

    """
    sleep 1
    """
}

workflow {

    myvals = Channel.of( 'A'..'Z' )

    test( myvals )
}

运行与：

nextflow run -ansi-log false main.nf

结果：

N E X T F L O W  ~  version 21.04.3
Launching `main.nf` [ecstatic_visvesvaraya] - revision: d34a24fe4a
[b9/21ef9b] Submitted process > test (myval: B)
[39/783aaf] Submitted process > test (myval: H)
[b5/aeae8b] Submitted process > test (myval: A)
[7a/e36e72] Submitted process > test (myval: D)
[de/25f001] Submitted process > test (myval: I)
[72/f913f2] Submitted process > test (myval: C)
[43/a2c78e] Submitted process > test (myval: L)
[5b/7c0434] Submitted process > test (myval: F)
[25/884e7c] Submitted process > test (myval: E)
[16/4c3b41] Submitted process > test (myval: G)
[62/c7bee1] Submitted process > test (myval: Q)
[71/cdbd37] Submitted process > test (myval: J)
[e6/634461] Submitted process > test (myval: N)
[37/03dd88] Submitted process > test (myval: S)
[fd/70867c] Submitted process > test (myval: K)
[cf/fb7e83] Submitted process > test (myval: T)
[56/3d6d41] Submitted process > test (myval: M)
[1e/81ad89] Submitted process > test (myval: O)
[db/66a292] Submitted process > test (myval: R)
[d5/212940] Submitted process > test (myval: Z)
[a8/1a33ab] Submitted process > test (myval: P)
[7e/60daa7] Submitted process > test (myval: U)
[d5/4d19c4] Submitted process > test (myval: V)
[13/8404ff] Submitted process > test (myval: W)
[22/adb044] Submitted process > test (myval: X)
[65/21a22b] Submitted process > test (myval: Y)

Nextflow，限制 Google 中的并发工作人员数量

Nextflow, Limit the number of concurrent workers in Google

nextflow