Ubuntu 升级后 运行 奇点时出现 squashfs 错误

squashfs error when running singularity after Ubuntu upgrade

概览:

我正在 运行ning Ubuntu 18.04 机器上处理奇点图像中的代码。截至周二晚上,它在机器 A 上运行(好吧,我正在调试代码)。我 运行 更新了,从第二天开始,任何对 运行 any 奇点容器的尝试都给我以下错误:

ERROR  : Failed to mount squashfs image in (read only): Invalid argument
ABORT  : Retval = 255

确认这不是我的代码造成的(通过 运行ning singularity exec <my_image_file>.sif 并得到同样的错误,我记得我在另一台机器上有几乎相同的设置(B)。我能够运行那里的图像没有任何问题。

不幸的是,我忘记了我在那台机器上选择了“无人值守更新”并重新启动。现在 returns 同样的错误。

我的尝试:

我这里有点空虚。我所做的只是说服自己,它与我的代码没有直接关系,而且几乎可以肯定与更新的 something 相关。我真的对 squashfs 知之甚少,而且对 singularity 还比较陌生。我还没有在网上找到任何有这个确切错误的东西(“无效参数”部分)。作为万福玛丽,我在机器 A 上重新安装了奇点,但它没有改变。

我不认为这是奇点本身的问题,而且我使用的是旧版本,所以我认为这不是一个有效的 github 问题。我使用的是 2.5.2 版,因为这是此代码适用于集群的版本。

详情

-vv 来自奇点的日志(注意:无论我传递什么参数,反馈都是相同的):

Increasing verbosity level (4)
Singularity version: 2.5.2-dist
Exec'ing: /usr/local/libexec/singularity/cli/exec.exec
Evaluating args: '<my_image_file>.sif <my_program> <program_arg>'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /usr/local/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'COMMAND' = 'exec'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
VERBOSE: Adding value to registry: 'VERSION' = '2.5.2-dist'
VERBOSE: Adding value to registry: 'IMAGE' = '<my_image_file>.sif'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'BINDIR' = '/usr/local/bin'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
VERBOSE: Set home (via getpwuid()) to: <my_home_directory>
VERBOSE: Running SUID program workflow
VERBOSE: Checking program has appropriate permissions
VERBOSE: Checking configuration file is properly owned by root
VERBOSE: Checking if singularity.conf allows us to run as suid
VERBOSE: Invoking the user namespace
VERBOSE: Not virtualizing USER namespace: running as SUID
VERBOSE: No autofs bug path in configuration, skipping
VERBOSE: Instantiating read only container image object
VERBOSE: Checking that file pointer is a Singularity image
VERBOSE: File is a valid SquashFS image
VERBOSE: Adding value to registry: 'CLEANUPD_FD' = '-1'
VERBOSE: Not virtualizing IPC namespace on user request
VERBOSE: Not virtualizing PID namespace on user request
VERBOSE: Not virtualizing network namespace on user request
VERBOSE: Using session directory: /usr/local/var/singularity/mnt/session
VERBOSE: Adding value to registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session'
VERBOSE: Found available loop device: /dev/loop18
VERBOSE: Using loop device: /dev/loop18
VERBOSE: Mounting squashfs image: /dev/loop18 -> /usr/local/var/singularity/mnt/container
ERROR  : Failed to mount squashfs image in (read only): Invalid argument
ABORT  : Retval = 255

grep for 'upgrade' in dpkg.log:

2020-07-23 06:10:36 upgrade evolution-data-server-common:all 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:10:41 upgrade linux-libc-dev:amd64 4.15.0-111.112 4.15.0-112.113
2020-07-23 06:10:46 upgrade python-pil:amd64 5.1.0-1ubuntu0.2 5.1.0-1ubuntu0.3
2020-07-23 06:10:51 upgrade python3-pil:amd64 5.1.0-1ubuntu0.2 5.1.0-1ubuntu0.3
2020-07-23 06:10:56 upgrade python2.7:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:57 upgrade libpython2.7:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:57 upgrade libpython2.7-stdlib:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:58 upgrade python2.7-minimal:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:58 upgrade libpython2.7-minimal:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade python3.6-venv:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade python3.6-dev:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade libpython3.6-dev:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade libpython3.6:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade python3.6:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade libpython3.6-stdlib:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:12 upgrade python3.6-minimal:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:12 upgrade libpython3.6-minimal:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:29 upgrade linux-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:11:29 upgrade linux-image-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:11:41 upgrade linux-headers-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:12:37 upgrade evolution-data-server:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:37 upgrade libebook-1.2-19:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedata-book-1.2-25:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedata-cal-1.2-28:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libebook-contacts-1.2-2:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libebackend-1.2-10:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libecal-1.2-19:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedataserver-1.2-23:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:39 upgrade libcamel-1.2-61:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:44 upgrade libavformat-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:44 upgrade libavcodec-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswresample-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswresample2:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswscale-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswscale4:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavresample-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavresample3:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavutil-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavutil55:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavcodec57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:47 upgrade libavformat57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:52 upgrade libpostproc54:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:57 upgrade libavfilter6:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:13:02 upgrade libavdevice57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:13:06 upgrade ffmpeg:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2

该图像是根据 Docker 图像构建的,并经过一些额外的步骤。我没有提供任何相关信息,因为行为似乎并不取决于我何时构建图像。

我遇到了这个确切的问题,我设法在 Ubuntu 18.04 中使用奇点版本 2.6.1 修复了它(不过我认为在 2.5.2 中解决方案对你来说不会有太大不同)。

要采取的步骤

  1. 如果需要请卸载早期版本的 Singularity。

  2. 通过打开终端和运行以下命令下载正确的 Singularity 版本的 tarball:

# Go to the place you want to work in
cd ~
VERSION=2.6.1  # Set an environment variable just for convenience, could replace with another version.
# Download
wget https://github.com/singularityware/singularity/releases/download/$VERSION/singularity-$VERSION.tar.gz
# Extract
tar xvf singularity-$VERSION.tar.gz
  1. 现在,应该有一个文件夹叫singularity-2.6.1(我是放在首页(~)。然后打开文件mount.c放在singularity-2.6.1/src/lib/image/squashfs/mount.c(我不确定 v2.5.2 是否相同,但我认为变化不大。不过我们可能运气不好)

  2. 在此文件中找到代码行

if ( singularity_mount(loop_dev, mount_point, "squashfs", MS_NOSUID|MS_RDONLY|MS_NODEV, "errors=remount-ro") < 0 )

改为

if ( singularity_mount(loop_dev, mount_point, "squashfs", MS_NOSUID|MS_RDONLY|MS_NODEV, "") < 0 )

别忘了保存!

  1. Return 到终端,通过 运行 命令
  2. 从编辑的安装文件安装 Singularity
cd singularity-$VERSION
./configure --prefix=/usr/local
make
sudo make install

对我来说,问题现在已经解决了!

编辑

正如 aguadopd 所指出的,如果您没有安装 libarchive-devmake 命令可能无法工作。如果是这种情况,那么首先在终端 运行 中修复它

sudo apt-get update -y
sudo apt-get install -y libarchive-dev

然后在一个新终端

singularity shell -p $HOME/Downloads/hrwros-09.simg

问题原理

显然,squashfs 中曾经有一个错误,如果它不知道,它就会简单地忽略参数。当今年早些时候修复此问题时,它会在遇到无效参数时开始出错,就像我们的案例一样。 (这里也有解释https://www.mail-archive.com/debian-kernel@lists.debian.org/msg117435.html) 因此,删除未知参数 "errors=remount-ro" 解决了问题。

老实说,由于我是 100% 的新手,所以我不确定它是否会在其他地方引起任何新的错误。然而,这也是在后来的 Singularity (https://github.com/hpcng/singularity/pull/4802/files/3f3136be150d9a8f63e58a0b4e39dc8487f1c0dc) 版本中基本修复此错误的方式。

希望对您有所帮助!

谢谢。我按照你的步骤做了,成功了。 但是,我在“make”步骤中遇到了问题。它没有用,因为我没有安装“libarchive-dev”。为了修复它,我做了

  1. sudo apt-get update -y
  2. sudo apt-get install -y libarchive-dev

然后在一个新终端

  1. 奇点shell -p $HOME/Downloads/hrwros-09.simg

现在可以使用了。