云数据流中的最大并发作业数

Maximum number of concurrent jobs in cloud dataflow

并发数据流作业(不是核心)的数量是否有限制? 我试图同时向 运行 提交约 40 个小作业,但在 7 个作业后开始出现 429 Rate Limit Exceeded 错误。

谢谢, G

是的,每个项目的并发作业数有限制。

在 alpha 版中,我们目前支持 10 个并发作业。如果您希望在 alpha 期间增加此限制,请联系 dataflow-feedback [at] google [dot] com.

如官方文档所述here,

You may run up to 25 concurrent Dataflow jobs per Cloud Platform project.

The Dataflow service is currently limited to processing job requests that are 10MB in size or smaller. The size of the job request is specifically tied to the JSON representation of your pipeline; a larger pipeline means a larger request.