如何让多个 MLFlow 并行运行？

Question

我对 Python 中的并行化不是很熟悉，尝试在多个训练折叠上并行训练模型时遇到错误。这是我的代码的简化版本：

def train_test_model(fold):
    # here I train the model etc...
    
    # now I want to save the parameters and metrics
    with mlflow.start_run():
        mlflow.log_param("run_name", run_name)
        mlflow.log_param("modeltype", modeltype)
        # and so on...

if __name__=="__main__":
    pool = ThreadPool(processes = num_trials)
    # run folds in parallel
    pool.map(lambda fold:train_test_model(fold), folds)

我收到以下错误：

Exception: Run with UUID 23e9bb6d22674a518e48af9c51252860 is already active. To start a new run, first end the current run with mlflow.end_run(). To start a nested run, call start_run with nested=True

documentation 说 mlflow.start_run() 启动了一个新的运行并使其处于活动状态，这是我问题的根源。每个线程为其相应的折叠启动一个 MLFlow 运行并使其处于活动状态，而我需要运行s 到运行并行，即全部处于活动状态（？）并保存 parameters/metrics对应的折叠。我该如何解决这个问题？

Answer 1

我找到了解决方案，也许对其他人有用。您可以在此处查看代码示例的详细信息：https://github.com/mlflow/mlflow/issues/3592

如何让多个 MLFlow 并行运行？

How to have multiple MLFlow runs in parallel?

python

parallel-processing

pyspark

mlflow