snakemake：使用运行指令时如何实现日志指令？

Question

Snakemake 允许使用指定日志文件名称的 log 参数为每个规则创建日志。将 shell 输出的结果通过管道传输到此日志相对简单，但我无法找到一种记录 run 输出（即 python 脚本）的方法。

一种解决方法是将 python 代码保存在脚本中，然后从 shell 中保存运行它，但我想知道是否还有其他方法？

Answer 1

我有一些规则同时使用 log 和 run 指令。在 run 指令中，我 "manually" 打开并写入日志文件。

例如：

rule compute_RPM:
    input:
        counts_table = source_small_RNA_counts,
        summary_table = rules.gather_read_counts_summaries.output.summary_table,
        tags_table = rules.associate_small_type.output.tags_table,
    output:
        RPM_table = OPJ(
            annot_counts_dir,
            "all_{mapped_type}_on_%s" % genome, "{small_type}_RPM.txt"),
    log:
        log = OPJ(log_dir, "compute_RPM_{mapped_type}", "{small_type}.log"),
    benchmark:
        OPJ(log_dir, "compute_RPM_{mapped_type}", "{small_type}_benchmark.txt"),
    run:
        with open(log.log, "w") as logfile:
            logfile.write(f"Reading column counts from {input.counts_table}\n")
            counts_data = pd.read_table(
                input.counts_table,
                index_col="gene")
            logfile.write(f"Reading number of non-structural mappers from {input.summary_table}\n")
            norm = pd.read_table(input.summary_table, index_col=0).loc["non_structural"]
            logfile.write(str(norm))
            logfile.write("Computing counts per million non-structural mappers\n")
            RPM = 1000000 * counts_data / norm
            add_tags_column(RPM, input.tags_table, "small_type").to_csv(output.RPM_table, sep="\t")

对于写入 stdout 的第三方代码，redirect_stdout 上下文管理器可能会有所帮助（在中找到，记录在 https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout).

测试蛇文件，test_run_log.snakefile:

from contextlib import redirect_stdout

rule all:
    input:
        "test_run_log.txt"

rule test_run_log:
    output:
        "test_run_log.txt"
    log:
        "test_run_log.log"
    run:
        with open(log[0], "w") as log_file:
            with redirect_stdout(log_file):
                print(f"Writing result to {output[0]}")
                with open(output[0], "w") as out_file:
                    out_file.write("result\n")

运行它：

$ snakemake -s test_run_log.snakefile

结果：

$ cat test_run_log.log 
Writing result to test_run_log.txt
$ cat test_run_log.txt 
result

Answer 2

我的解决方案如下。这对于普通日志和带有回溯的日志记录异常都很有用。然后，您可以将记录器设置包装在一个函数中，使其更有条理。虽然它不是很漂亮。如果snakemake能自己做就更好了

import logging

# some stuff

rule logging_test:
    input: 'input.json'
    output: 'output.json'
    log: 'rules_logs/logging_test.log'
    run:
        logger = logging.getLogger('logging_test')
        fh = logging.FileHandler(str(log))
        fh.setLevel(logging.INFO)
        formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
        fh.setFormatter(formatter)
        logger.addHandler(fh)

        try: 
            logger.info('Starting operation!')
            # do something
            with open(str(output), 'w') as f:
                f.write('success!')
            logger.info('Ended!')
        except Exception as e: 
            logger.error(e, exc_info=True)

snakemake：使用 运行 指令时如何实现日志指令？

snakemake: how to implement log directive when using run directive?

snakemake

snakemake：使用运行指令时如何实现日志指令？