Tf-Agents ParallelPyEnvironment 静默失败

Question

我已经编写了一个自定义环境，因此我可以使用强化学习 (PPO) 和 tf-agents。如果我将我的环境（从 py_environment.PyEnvironment 继承）包装在 TfPyEnvironment 中，这会很好地工作，但如果我尝试将它包装成 ParallelPyEnvironment，则会失败。我试过使用 ParallelPyEnvironment 的所有关键字参数，但代码只是运行到该行然后什么也没有发生 - 没有异常，程序没有终止等

这是我的代码，用于初始化环境并展示 eval_env 的工作变体：

train_env = tf_py_environment.TFPyEnvironment(
    ParallelPyEnvironment(
        [CardGameEnv()] * hparams['parallel_environments']
    )
)
# this works perfectly:
eval_env = tf_py_environment.TFPyEnvironment(CardGameEnv(debug=True))

如果我通过 CTRL+C 终止脚本，这就是输出的内容：

Traceback (most recent call last):
Traceback (most recent call last):
  File "E:\Users\tmp\Documents\Programming\Neural Nets\Poker_AI\poker_logic\train.py", line 229, in <module>
  File "<string>", line 1, in <module>
    train(model_num=3)
  File "C:\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "E:\Users\tmp\Documents\Programming\Neural Nets\Poker_AI\poker_logic\train.py", line 64, in train
    [CardGameEnv()] * hparams['parallel_environments']
    exitcode = _main(fd)
  File "E:\Users\tmp\AppData\Roaming\Python\Python37\site-packages\gin\config.py", line 1009, in wrapper
  File "C:\Python37\lib\multiprocessing\spawn.py", line 113, in _main
    preparation_data = reduction.pickle.load(from_parent)
KeyboardInterrupt
    return fn(*new_args, **new_kwargs)
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 70, in __init__
    self.start()
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 83, in start
    env.start(wait_to_start=self._start_serially)
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 223, in start
    self._process.start()
  File "C:\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Python37\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 264, in __getattr__
    return self._receive()
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 333, in _receive
    message, payload = self._conn.recv()
  File "C:\Python37\lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Python37\lib\multiprocessing\connection.py", line 306, in _recv_bytes
    [ov.event], False, INFINITE)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 289, in close
    self._process.join(5)
  File "C:\Python37\lib\multiprocessing\process.py", line 139, in join
    assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process

由此我得出结论，线程 ParallelPyEnvironment 正在尝试启动并没有这样做，但由于我对 Python 中的线程处理经验不是很丰富，我不知道该去哪里从这里开始，尤其是如何解决这个问题。当前的训练需要很长时间并且根本不使用我的 PC 的功能（使用 3GB 的 32GB RAM，处理器为 3%，GPU 几乎不工作，但 VRAM 已满），所以这应该会显着加快训练时间。

Answer 1

解决方案是传入可调用对象，而不是环境，因此 ParallelPyEnvironment 可以自己构造它们：

train_env = tf_py_environment.TFPyEnvironment(
    ParallelPyEnvironment(
        [CardGameEnv] * hparams['parallel_environments']
    )
)

Tf-Agents ParallelPyEnvironment 静默失败

Tf-Agents ParallelPyEnvironment fails silently

python

python-multithreading

tensorflow

tensorflow-agents