SCons 中 chdir=1 和 num_jobs>1 的组合

Question

我有一个相当冗长的测试任务，它由 SCons 自动执行并且可以并行化。但是，它目前依赖于使用 chdir=1，目前删除它并不容易。现在，一旦我分别使用 -j2 SetOption('num_jobs', 2) 作业就会失败，下面的最小（非）工作示例说明了原因：chdir 不是按线程应用的，而是会影响所有作业一次。

这种行为是故意的吗？有什么办法可以防止这种情况发生吗？

# File SConstruct
import os, time

def my_build_fun(target, source, env):
    for i in range(1, 5):
        out = "my_build_fun: %d %s %s" % (i, str(source[0]), os.getcwd())
        print out
        time.sleep(0.5)
    return None

bld = Builder(action = my_build_fun,
              suffix = '.output',
              src_suffix = '.input',
              chdir=1)
env = Environment(BUILDERS = {'Foo' : bld})

Alias('do_a', env.Foo('folder_a/do_a'))
Alias('do_b', env.Foo('folder_b/do_b'))

Default(['do_a', 'do_b'])

在

上运行

+--- SConstruct
+-+- folder_a
| \--- do_a.input
\-+- folder_b
  \--- do_b.input

scons -j1 的结果：符合预期

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
os.chdir('folder_a')
my_build_fun(["folder_a\do_a.output"], ["folder_a\do_a.input"])
my_build_fun: 1 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 2 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 3 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 4 folder_a\do_a.input H:\Playground\folder_a
os.chdir('H:\Playground')
os.chdir('folder_b')
my_build_fun(["folder_b\do_b.output"], ["folder_b\do_b.input"])
my_build_fun: 1 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 2 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 3 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 4 folder_b\do_b.input H:\Playground\folder_b
os.chdir('H:\Playground')
scons: done building targets.

scons -j2 的结果：尝试将 chdir 放入 .\folder_a\folder_b，但实际上并不存在。

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
os.chdir('folder_a')
my_build_fun(["folder_a\do_a.output"], ["folder_a\do_a.input"])
my_build_fun: 1 folder_a\do_a.input H:\Playground\folder_a
os.chdir('folder_b')
my_build_fun(["folder_b\do_b.output"], ["folder_b\do_b.input"])
scons: *** [folder_b\do_b.output] folder_b: The system cannot find the file specified
my_build_fun: 2 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 3 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 4 folder_a\do_a.input H:\Playground\folder_a
my_build_fun
os.chdir('H:\Playground')
scons: building terminated because of errors.

Answer 1

我不知道它是否有意，但工作目录不是线程安全的。

我遇到过一次这个问题，解决它的唯一方法是不更改目录，而是使用相对 and/or 绝对路径。

SCons 邮件档案中有这方面的信息，它们可能会为您提供更多上下文。

scons-users@scons.org

Answer 2

来自 SCons 手册页

http://www.scons.org/doc/2.3.2/HTML/scons-man.html

WARNING: Python only keeps one current directory location for all of the threads. This means that use of the chdir argument will not work with the SCons -j option, because individual worker threads spawned by SCons interfere with each other when they start changing directory.

因此，如果您的脚本不能运行来自构建的基本目录。解决这个问题的（当前）最简单的方法是创建一个脚本，它将 cd 然后运行你的命令。或者甚至把它放在 shell 命令中执行

env.Command(b,a,"cd ${SOURCE.dir}; do whatever -o ${TARGET.file} -i ${SOURCE.file}").

SCons 中 chdir=1 和 num_jobs>1 的组合

Combination of chdir=1 and num_jobs>1 in SCons

python

build

scons

multiprocessing