在需要在导入过程中创建文件的 Singularity 容器中安装 Python 包（例如启用 GPU 渲染的 mujoco_py）

Question

请注意，以下部分描述特定于 Python 包 mujoco_py，但问题本身是普遍的。

我按照 here and here 中描述的步骤构建了一个容器，其中安装了 mujoco-py，使用 GPU 进行渲染。但是，当 import mujoco_py 时，当 mujoco-py 尝试创建一些使用 GPU 渲染所需的 files/directories 时，我会收到以下错误。但是，由于文件系统变为只读，因此在构建容器后无法在容器内创建任何文件，除非这些文件将存储在 /tmp 中。此外，就我而言，将容器构建为可写容器不是一个选项。我还尝试通过 python3 -m pip install . -e 而不是 python3 setup.py install 安装 mujoco-py 但是当我安装 import mujoco_py.

时出现了一些其他类似的错误

有没有办法更改将要创建这些文件的路径或将这些文件保留在内存中？

import mujoco_py
running build_ext
building 'mujoco_py.cymj' extension
creating /usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/generated/_pyxbld_2.0.2.13_37_linuxgpuextensionbuilder
Traceback (most recent call last):
  File "/usr/lib/python3.7/distutils/dir_util.py", line 70, in mkpath
    os.mkdir(head, mode)
OSError: [Errno 30] Read-only file system: '/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/generated/_pyxbld_2.0.2.13_37_linuxgpuextensionbuilder'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/__init__.py", line 3, in <module>
    from mujoco_py.builder import cymj, ignore_mujoco_warnings, functions, MujocoException
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 510, in <module>
    cymj = load_cython_ext(mujoco_path)
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 105, in load_cython_ext
    cext_so_path = builder.build()
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 221, in build
    built_so_file_path = self._build_impl()
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 291, in _build_impl
    so_file_path = super()._build_impl()
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 244, in _build_impl
    dist.run_commands()
  File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.7/dist-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/builder.py", line 144, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/local/lib/python3.7/dist-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/usr/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/usr/lib/python3.7/distutils/ccompiler.py", line 566, in compile
    depends, extra_postargs)
  File "/usr/lib/python3.7/distutils/ccompiler.py", line 348, in _setup_compile
    self.mkpath(os.path.dirname(obj))
  File "/usr/lib/python3.7/distutils/ccompiler.py", line 916, in mkpath
    mkpath(name, mode, dry_run=self.dry_run)
  File "/usr/lib/python3.7/distutils/dir_util.py", line 74, in mkpath
    "could not create '%s': %s" % (head, exc.args[-1]))
distutils.errors.DistutilsFileError: could not create '/usr/local/lib/python3.7/dist-packages/mujoco_py-2.0.2.13-py3.7.egg/mujoco_py/generated/_pyxbld_2.0.2.13_37_linuxgpuextensionbuilder': Read-only file system

这就是我的构建方式

Answer 1

现在可以通过 --overlay 在 Singularity 容器中写入 [临时] 文件到容器中：

https://sylabs.io/guides/3.7/user-guide/persistent_overlays.html

在需要在导入过程中创建文件的 Singularity 容器中安装 Python 包（例如启用 GPU 渲染的 mujoco_py）

Installing a Python package (e.g. mujoco_py with GPU rendering enabled) in a Singularity container that requires creating files during import

python

gpu

pip

singularity-container

mujoco