为特定目标禁用并行构建
Disable paralled build for a specific target
我需要为单个目标禁用并行 运行。这是一个验证程序是否不会创建一些随机或错误命名文件的测试。同时构建的任何其他文件都无法通过此测试。
我在 SCons FAQ 上找到了这条建议：
Use the SideEffect() method and specify the same dummy file for each target that shouldn't be built in parallel. Even if the file doesn't exist, SCons will prevent the simultaneous execution of commands that affect the dummy file. See the linked method page for examples.
然而，这是无用的，因为它会阻止任何两个目标的并行构建，而不仅仅是测试脚本。
有没有什么方法可以阻止一个目标的并行构建，同时允许所有其他目标并行构建？
我们在 scons discord 中讨论了这个问题，并提出了一个示例，它将设置同步测试 运行ners，这将确保在测试 [=] 时没有其他任务正在 运行ning 16=].
这是来自 github example repo:
的示例 SConstruct
import SCons
# A bound map of stream (as in stream of work) name to side-effect
# file. Since SCons will not allow tasks with a shared side-effect
# to execute concurrently, this gives us a way to limit link jobs
# independently of overall SCons concurrency.
node_map = dict()
# A list of nodes that have to be run synchronously.
# sync node ensures the test runners are syncrhonous amongst
# themselves.
sync_nodes = list()
# this emitter will make a phony sideeffect per target
# the test builders will share all the other sideeffects making
# sure the tests only run when nothing else is running.
def sync_se_emitter(target, source, env):
name = str(target[0])
se_name = "#unique_node_" + str(hash(name))
se_node = node_map.get(se_name, None)
if not se_node:
se_node = env.Entry(se_name)
# This may not be necessary, but why chance it
env.NoCache(se_node)
node_map[se_name] = se_node
for sync_node in sync_nodes:
env.SideEffect(se_name, sync_node)
env.SideEffect(se_node, target)
return (target, source)
# here we force all builders to use the emitter, so all
# targets will respect the shared sideeffect when being built.
# NOTE: that the builders which should be synchronous must be listed
# by name, as SynchronousTestRunner is in this example
original_create_nodes = SCons.Builder.BuilderBase._create_nodes
def always_emitter_create_nodes(self, env, target = None, source = None):
if self.get_name(env) != "SynchronousTestRunner":
if self.emitter:
self.emitter = SCons.Builder.ListEmitter([self.emitter, sync_se_emitter])
else:
self.emitter = SCons.Builder.ListEmitter([sync_se_emitter])
return original_create_nodes(self, env, target, source)
SCons.Builder.BuilderBase._create_nodes = always_emitter_create_nodes
env = Environment()
env.Tool('textfile')
nodes = []
# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SCons.Builder.Builder(
action=SCons.Action.Action([
"sleep 1",
"echo Starting test $TARGET",
"sleep 5",
"echo Finished test $TARGET",
'echo done > $TARGET'],
None))
# this emitter connects the test runners with the shared sideeffect
def sync_test_emitter(target, source, env):
for name in node_map:
env.SideEffect(name, target)
sync_nodes.append(target)
return (target, source)
env['BUILDERS']["SynchronousTestRunner"].emitter = SCons.Builder.ListEmitter([sync_test_emitter])
# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")
for i in range(50):
nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))
for i in range(50, 76):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test.out")
nodes.append(node)
for i in range(76, 100):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test2.out")
nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))
env.Program('out', nodes)
此解决方案基于 dmoody256 的回答。
基本概念是相同的，但代码应该更易于使用，并且可以将其放在 site_scons 目录中，以免混淆 SConstruct 本身。
site_scons/site_init.py:
# Allows using functions `SyncBuilder` and `Environment.SyncCommand`.
from SyncBuild import SyncBuilder
site_scons/SyncBuild.py:
from SCons.Builder import Builder, BuilderBase, ListEmitter
from SCons.Environment import Base as BaseEnvironment
# This code allows to build some targets synchronously, which means there won't
# be anything else built at the same time even if SCons is run with flag `-j`.
#
# This is achieved by adding a different dummy values as side effect of each
# target. (These files won't be created. They are only a way of enforcing
# constraints on SCons.)
# Then the files that need to be built synchronously have added every dummy
# value from the entire configuration as a side effect, which effectively
# prevents it from being built along with any other file.
#
# To create a synchronous target use `SyncBuilder`.
__processed_targets = set()
__lock_values = []
__synchronous_nodes = []
def __add_emiter_to_builder(builder, emitter):
if builder.emitter:
builder.emitter = ListEmitter([builder.emitter, emitter])
else:
builder.emitter = emitter
def __inividual_sync_locks_emiter(target, source, env):
if not target or target[0] not in __processed_targets:
lock_value = env.Value(f'.#sync_lock_{len(__lock_values)}#')
env.NoCache(lock_value)
env.SideEffect(lock_value, target + __synchronous_nodes)
__processed_targets.update(target)
__lock_values.append(lock_value)
return target, source
__original_create_nodes = BuilderBase._create_nodes
def __create_nodes_adding_emiter(self, *args, **kwargs):
__add_emiter_to_builder(self, __inividual_sync_locks_emiter)
return __original_create_nodes(self, *args, **kwargs)
BuilderBase._create_nodes = __create_nodes_adding_emiter
def _all_sync_locks_emitter(target, source, env):
env.SideEffect(__lock_values, target)
__synchronous_nodes.append(target)
return (target, source)
def SyncBuilder(*args, **kwargs):
"""It works like the normal `Builder` except it prevents the targets from
being built at the same time as any other target."""
target = Builder(*args, **kwargs)
__add_emiter_to_builder(target, _all_sync_locks_emitter)
return target
def __SyncBuilder(self, *args, **kwargs):
"""It works like the normal `Builder` except it prevents the targets from
being built at the same time as any other target."""
target = self.Builder(*args, **kwargs)
__add_emiter_to_builder(target, _all_sync_locks_emitter)
return target
BaseEnvironment.SyncBuilder = __SyncBuilder
def __SyncCommand(self, *args, **kwargs):
"""It works like the normal `Command` except it prevents the targets from
being built at the same time as any other target."""
target = self.Command(*args, **kwargs)
_all_sync_locks_emitter(target, [], self)
return target
BaseEnvironment.SyncCommand = __SyncCommand
SConstruct（这是改编自 dmoody256 的测试，与原始测试做同样的事情）：
env = Environment()
env.Tool('textfile')
nodes = []
# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SyncBuilder(
action=Action([
"sleep 1",
"echo Starting test $TARGET",
"sleep 5",
"echo Finished test $TARGET",
'echo done > $TARGET'],
None))
# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")
for i in range(50):
nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))
for i in range(50, 76):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test.out")
nodes.append(node)
for i in range(76, 100):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test2.out")
nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))
env.Program('out', nodes)
创建 site_scons/site_init.py 和 site_scons/SyncBuild.py 后，您可以在任何 SConstruct 或 SConscript 文件中使用函数 SyncBuilder 或方法 Environment.SyncCommand在项目中无需任何额外配置。
我需要为单个目标禁用并行 运行。这是一个验证程序是否不会创建一些随机或错误命名文件的测试。同时构建的任何其他文件都无法通过此测试。
我在 SCons FAQ 上找到了这条建议：
Use the SideEffect() method and specify the same dummy file for each target that shouldn't be built in parallel. Even if the file doesn't exist, SCons will prevent the simultaneous execution of commands that affect the dummy file. See the linked method page for examples.
然而，这是无用的，因为它会阻止任何两个目标的并行构建，而不仅仅是测试脚本。
有没有什么方法可以阻止一个目标的并行构建，同时允许所有其他目标并行构建？
我们在 scons discord 中讨论了这个问题，并提出了一个示例，它将设置同步测试 运行ners，这将确保在测试 [=] 时没有其他任务正在 运行ning 16=].
这是来自 github example repo:
的示例 SConstructimport SCons
# A bound map of stream (as in stream of work) name to side-effect
# file. Since SCons will not allow tasks with a shared side-effect
# to execute concurrently, this gives us a way to limit link jobs
# independently of overall SCons concurrency.
node_map = dict()
# A list of nodes that have to be run synchronously.
# sync node ensures the test runners are syncrhonous amongst
# themselves.
sync_nodes = list()
# this emitter will make a phony sideeffect per target
# the test builders will share all the other sideeffects making
# sure the tests only run when nothing else is running.
def sync_se_emitter(target, source, env):
name = str(target[0])
se_name = "#unique_node_" + str(hash(name))
se_node = node_map.get(se_name, None)
if not se_node:
se_node = env.Entry(se_name)
# This may not be necessary, but why chance it
env.NoCache(se_node)
node_map[se_name] = se_node
for sync_node in sync_nodes:
env.SideEffect(se_name, sync_node)
env.SideEffect(se_node, target)
return (target, source)
# here we force all builders to use the emitter, so all
# targets will respect the shared sideeffect when being built.
# NOTE: that the builders which should be synchronous must be listed
# by name, as SynchronousTestRunner is in this example
original_create_nodes = SCons.Builder.BuilderBase._create_nodes
def always_emitter_create_nodes(self, env, target = None, source = None):
if self.get_name(env) != "SynchronousTestRunner":
if self.emitter:
self.emitter = SCons.Builder.ListEmitter([self.emitter, sync_se_emitter])
else:
self.emitter = SCons.Builder.ListEmitter([sync_se_emitter])
return original_create_nodes(self, env, target, source)
SCons.Builder.BuilderBase._create_nodes = always_emitter_create_nodes
env = Environment()
env.Tool('textfile')
nodes = []
# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SCons.Builder.Builder(
action=SCons.Action.Action([
"sleep 1",
"echo Starting test $TARGET",
"sleep 5",
"echo Finished test $TARGET",
'echo done > $TARGET'],
None))
# this emitter connects the test runners with the shared sideeffect
def sync_test_emitter(target, source, env):
for name in node_map:
env.SideEffect(name, target)
sync_nodes.append(target)
return (target, source)
env['BUILDERS']["SynchronousTestRunner"].emitter = SCons.Builder.ListEmitter([sync_test_emitter])
# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")
for i in range(50):
nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))
for i in range(50, 76):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test.out")
nodes.append(node)
for i in range(76, 100):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test2.out")
nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))
env.Program('out', nodes)
此解决方案基于 dmoody256 的回答。
基本概念是相同的，但代码应该更易于使用，并且可以将其放在 site_scons 目录中，以免混淆 SConstruct 本身。
site_scons/site_init.py:
# Allows using functions `SyncBuilder` and `Environment.SyncCommand`.
from SyncBuild import SyncBuilder
site_scons/SyncBuild.py:
from SCons.Builder import Builder, BuilderBase, ListEmitter
from SCons.Environment import Base as BaseEnvironment
# This code allows to build some targets synchronously, which means there won't
# be anything else built at the same time even if SCons is run with flag `-j`.
#
# This is achieved by adding a different dummy values as side effect of each
# target. (These files won't be created. They are only a way of enforcing
# constraints on SCons.)
# Then the files that need to be built synchronously have added every dummy
# value from the entire configuration as a side effect, which effectively
# prevents it from being built along with any other file.
#
# To create a synchronous target use `SyncBuilder`.
__processed_targets = set()
__lock_values = []
__synchronous_nodes = []
def __add_emiter_to_builder(builder, emitter):
if builder.emitter:
builder.emitter = ListEmitter([builder.emitter, emitter])
else:
builder.emitter = emitter
def __inividual_sync_locks_emiter(target, source, env):
if not target or target[0] not in __processed_targets:
lock_value = env.Value(f'.#sync_lock_{len(__lock_values)}#')
env.NoCache(lock_value)
env.SideEffect(lock_value, target + __synchronous_nodes)
__processed_targets.update(target)
__lock_values.append(lock_value)
return target, source
__original_create_nodes = BuilderBase._create_nodes
def __create_nodes_adding_emiter(self, *args, **kwargs):
__add_emiter_to_builder(self, __inividual_sync_locks_emiter)
return __original_create_nodes(self, *args, **kwargs)
BuilderBase._create_nodes = __create_nodes_adding_emiter
def _all_sync_locks_emitter(target, source, env):
env.SideEffect(__lock_values, target)
__synchronous_nodes.append(target)
return (target, source)
def SyncBuilder(*args, **kwargs):
"""It works like the normal `Builder` except it prevents the targets from
being built at the same time as any other target."""
target = Builder(*args, **kwargs)
__add_emiter_to_builder(target, _all_sync_locks_emitter)
return target
def __SyncBuilder(self, *args, **kwargs):
"""It works like the normal `Builder` except it prevents the targets from
being built at the same time as any other target."""
target = self.Builder(*args, **kwargs)
__add_emiter_to_builder(target, _all_sync_locks_emitter)
return target
BaseEnvironment.SyncBuilder = __SyncBuilder
def __SyncCommand(self, *args, **kwargs):
"""It works like the normal `Command` except it prevents the targets from
being built at the same time as any other target."""
target = self.Command(*args, **kwargs)
_all_sync_locks_emitter(target, [], self)
return target
BaseEnvironment.SyncCommand = __SyncCommand
SConstruct（这是改编自 dmoody256 的测试，与原始测试做同样的事情）：
env = Environment()
env.Tool('textfile')
nodes = []
# this is a fake test runner which acts like its running a test
env['BUILDERS']["SynchronousTestRunner"] = SyncBuilder(
action=Action([
"sleep 1",
"echo Starting test $TARGET",
"sleep 5",
"echo Finished test $TARGET",
'echo done > $TARGET'],
None))
# in this test we create two test runners and make them depend on various source files
# being generated. This is just to force the tests to be run in the middle of
# the build. This will allow the example to demonstrate that all other jobs
# have paused so the test can be performed.
env.SynchronousTestRunner("test.out", "source10.c")
env.SynchronousTestRunner("test2.out", "source62.c")
for i in range(50):
nodes.append(env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}"))
for i in range(50, 76):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test.out")
nodes.append(node)
for i in range(76, 100):
node = env.Textfile(f"source{i}.c", f"int func{i}(){{return {i};}}")
env.Depends(node, "test2.out")
nodes.append(node)
nodes.append(env.Textfile('main.c', 'int main(){return 0;}'))
env.Program('out', nodes)
创建 site_scons/site_init.py 和 site_scons/SyncBuild.py 后，您可以在任何 SConstruct 或 SConscript 文件中使用函数 SyncBuilder 或方法 Environment.SyncCommand在项目中无需任何额外配置。