嵌套的 Luigi 任务不会出现在执行摘要中

Question

我计划使用 Luigi 编写可重现的 failure-resistant hyperparameter-tuning 任务。因此，我在 "parent" class HParamOptimizer.

中多次调用 class TrainOneModel

为了简化这里的事情，这里有一个更简单的 hello world 版本：

import luigi

# Child class
class HelloTask(luigi.Task):
    name = luigi.parameter.Parameter(default='Luigi')

    def run(self):
        print(f'Luigi says: Hello {self.name}!')

# Parent class
class ManyHellos(luigi.Task):

    def run(self):
        names = ['Marc', 'Anna', 'John']
        for name in names:
            hello = HelloTask(name=name)
            hello.run()

if __name__ == '__main__':
    luigi.run(['ManyHellos', '--workers', '1', '--local-scheduler'])

运行带有 python filename.py 的脚本有效并且看起来有进展 :)。名称也按预期打印，但是，执行摘要仅显示 ManyHellos 运行:

Scheduled 1 tasks of which:
* 1 ran successfully:
    - 1 ManyHellos()

是否有可能包括 child class HelloTask 以在中央调度可视化器中查看事情的进展情况？

谢谢，BBQuercus

Answer 1

它们没有显示，因为您是手动执行它们的运行() 而不是通过调度程序。路易吉的做法更像这样：

import luigi


# Child class
class HelloTask(luigi.Task):
    name = luigi.parameter.Parameter(default='Luigi')

    def run(self):
        with self.output().open('w') as fout:
            fout.write(f'Luigi says: Hello {self.name}!')

    def output(self):
        # An output target is needed for the scheduler to verify whether
        # the task was run.
        return luigi.LocalTarget(f'./names/{self.name}.txt')


# Parent class
class ManyHellos(luigi.Task):
    def run(self):
        names = ['Marc', 'Anna', 'John']
        for name in names:
            yield HelloTask(name=name)  # dynamically schedules a HelloTask


if __name__ == '__main__':
    luigi.run(['ManyHellos', '--workers', '1', '--local-scheduler'])

这导致

===== Luigi Execution Summary =====

Scheduled 4 tasks of which:
* 4 ran successfully:
    - 3 HelloTask(name=Anna,John,Marc)
    - 1 ManyHellos()

This progress looks :) because there were no failed tasks or missing dependencies

执行时。请注意，您也可以一次产生多个任务，使用 yield [HelloTask(name) for name in names]

嵌套的 Luigi 任务不会出现在执行摘要中

Nested Luigi tasks do not appear in execution summary

python

workflow

luigi