最先进的作业调度(容器、混合云)?

State-of-the-art job scheduling (containers, hybrid cloud)?

我们有一种评估工作,它包括对具有各种输入的遗留二进制文件的数千次调用,每个 运行 就像一分钟。各个运行是完全可并行的(每个内核一个实例)。

在混合云场景中执行此操作的最新技术是什么?

Kubernetes 本身似乎没有提供用于确定等待作业优先级或管理等待作业的界面。詹金斯在这些方面会很擅长,但感觉就像一个 hack。当然,我们可以自己破解一些东西,但问题应该足够通用,以至于已经有了开箱即用的解决方案。

有很多框架可以帮助管理 Kubernetes 集群中的作业。最受欢迎的是:

  • Argo 用于在 Kubernetes 上编排并行作业。工作流作为 Kubernetes CRD(自定义资源定义)实现。
  • Airflow - has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Also take a look for kubernetes-executor

我建议您寻找 this video,它描述了每个框架并帮助您决定哪个更适合您。

您可能对以下有关使用 Mesos 进行混合云的文章感兴趣

Hybrid cloud technology is becoming increasingly popular as it merges private and public clouds to bring the best of two worlds together. However, due to the heterogeneous cloud installation, facilitating a hybrid cloud setup is not simple. Despite the availability of some commercial solutions to build a hybrid cloud, an open source implementation is still unavailable. In this paper, we try to bridge the gap by providing an open source implementation by leveraging the power of Apache Mesos. We build a hybrid cloud on the top of multiple cloud platforms, private and public.