Nextflow with Azure Batch - 找不到匹配的 VM 映像

Question

尝试使用 Azure Batch (NF-Core) 设置 Nextflow 时，出现以下错误。我在多个工作流程（sarek、ataseq 等）上尝试过这个。我得到了同样的错误 -

N E X T F L O W  ~  version 22.04.0
Pulling nf-core/atacseq ...
 downloaded from https://github.com/nf-core/atacseq.git
Launching `https://github.com/nf-core/atacseq` [rhl6d5529] DSL1 - revision: 1b3a832db5 [1.2.1]
Downloading plugin nf-azure@0.13.1
----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/atacseq v1.2.1
----------------------------------------------------

Run Name             : rhl6d5529
Data Type            : Paired-End
Design File          : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/design.csv
Genome               : Not supplied
Fasta File           : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genome.fa
GTF File             : https://raw.githubusercontent.com/nf-core/test-datasets/atacseq/reference/genes.gtf
Mitochondrial Contig : MT
MACS2 Genome Size    : 1.2E+7
Min Consensus Reps   : 1
MACS2 Narrow Peaks   : No
MACS2 Broad Cutoff   : 0.1
Trim R1              : 0 bp
Trim R2              : 0 bp
Trim 3' R1           : 0 bp
Trim 3' R2           : 0 bp
NextSeq Trim         : 0 bp
Fingerprint Bins     : 100
Save Genome Index    : No
Max Resources        : 6 GB memory, 2 cpus, 12h time per job
Container            : docker - nfcore/atacseq:1.2.1
Output Dir           : ./results
Launch Dir           : /
Working Dir          : /nextflow/atacseq/rhl6d5529
Script Dir           : /.nextflow/assets/nf-core/atacseq
User                 : root
Config Profile       : test,azurebatch
Config Description   : Minimal test dataset to check pipeline function
Config Contact       : Venkat Malladi (@vsmalladi)
Config URL           : https://azure.microsoft.com/services/batch/
----------------------------------------------------
Uploading local `bin` scripts folder to az://nextflow/atacseq/rhl6d5529/tmp/66/bd55d79e42999df38ba04a81c3aa04/bin
[-        ] process > CHECK_DESIGN -

[-        ] process > CHECK_DESIGN [  0%] 0 of 1

[-        ] process > CHECK_DESIGN [  0%] 0 of 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified




[58/55b7f7] process > CHECK_DESIGN (design.csv) [100%] 1 of 1, failed: 1
Error executing process > 'CHECK_DESIGN (design.csv)'

Caused by:
  Cannot find a matching VM image with publisher=microsoft-azure-batch; offer=centos-container; OS type=linux; verification type=verified

我试着查看 nextflow 的源代码。我发现错误在 AzBatchService.groovy（下面的行号）中。

https://github.com/nextflow-io/nextflow/blob/0e593e6ab82880810d8139a4fe6e3c47ff69a531/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy#L442

我在我的 Azure Batch 帐户实例中做了一些进一步的挖掘。基本上，我想确认从 Azure Batch 帐户收到的支持图像列表是否包含此管道所需的图像。我可以确认服务器确实响应了所需的图像 -

这可能是什么问题？我记得几周前运行完全相同的管道，它确实工作了几次。我错过了什么吗？

Answer 1

刚刚又看了一遍 Azure Cloud 文档，认为这可能是相关的：

By default, Nextflow creates CentOS 8-based pool nodes, but this behavior can be customised in the pool configuration. Below the configurations for image reference/SKU combinations to select two popular systems.

Ubuntu 20.04:
sku = "batch.node.ubuntu 20.04"
offer = "ubuntu-server-container"
publisher = "microsoft-azure-batch"
CentOS 8 (default):
sku = "batch.node.centos 8"
offer = "centos-container"
publisher = "microsoft-azure-batch"

我认为这里的问题是不匹配 nodeAgentSkuId。 Nextflow 期望 CentOS 8 节点代理 SKU，但您有 CentOS 7 SKU。如果无法以某种方式更改 nodeAgentSkuId，Nextflow 使用的节点代理 SKU 应该可以通过将其添加到您的 nextflow.config:

来覆盖

azure.batch.pools.<name>.sku = 'batch.node.centos 7'

其中 <name> 是池标识符：

azure.batch.pools.<name>.sku

Specify the ID of the Compute Node agent SKU which the pool identified with <name> supports (default: batch.node.centos 8, requires nf-azure@0.11.0).

https://www.nextflow.io/docs/edge/azure.html#advanced-settings

Nextflow with Azure Batch - 找不到匹配的 VM 映像

Nextflow with Azure Batch - Cannot find a matching VM image

azure

azure-batch

nextflow