docker-compose 突然出现 I/O 错误

Sudden I/O errors with docker-compose

我正在处理的项目有一个 docker-compose 开发环境,突然间多个容器开始出现 I/O 错误,如下所示:

Error: EIO: i/o error, open '/usr/app/src/components/common/FilterComponent.tsx'

NGINX 因同样原因崩溃,依此类推:

nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (5: Input/output error)

Docker 有足够的磁盘 space(使用了 112GB avail/32GB),我的 Mac 上的其他一切似乎工作正常。让它再次工作的唯一方法是 Docker 恢复出厂设置,直到它再次发生(~1-2 天)。

这是 docker-compose.yml 文件,如果你问我的话,这很基本:

x-env-files: &env-files
  env_file:
    - docker.env
version: '3.3'
services:
  mongo-store:
    image: mongo:latest
    volumes:
      - /data/db
    ports:
      - 27017:27017
    networks:
      - backend
    command: ['mongod', '--bind_ip', '0.0.0.0']
  application-ui:
    build:
      context: .
      dockerfile: Dockerfile-dev
    image: application-ui:dev
    ports:
      - 5000:5000
    networks:
      - frontend
    stdin_open: true
    volumes:
      - ./src/:/usr/app/src/
    command: ['npm', 'run', 'start']

  application-service:
    build:
      context: ../application-service
      dockerfile: Dockerfile-local
    image: application-service:dev
    ports:
      - 9000:9000
    networks:
      - backend
      - frontend
    environment:
      - 'GITLAB_TOKEN=${GITLAB_TOKEN}'
    <<: *env-files
    volumes:
      - ../application-service/src/:/usr/app/src/
  nginx:
    image: nginx:latest
    volumes:
      - ./nginx/nginx.local.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
      - 443:443
    networks:
      - backend
      - frontend
networks:
  backend:
  frontend:

这是docker run --rm -v /:/host busybox df -h的输出:

Filesystem                Size      Used Available Use% Mounted on
overlay                 149.4G     34.8G    106.9G  25% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.9G         0      3.9G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
overlay                   3.9G    316.0K      3.9G   0% /host
dev                       3.8G         0      3.8G   0% /host/dev
shm                       3.9G         0      3.9G   0% /host/dev/shm
/dev/vda1               149.4G     34.8G    106.9G  25% /host/etc/cni/net.d
/dev/vda1               149.4G     34.8G    106.9G  25% /host/etc/kubernetes
tmpfs                   796.0M    480.0K    795.5M   0% /host/etc/resolv.conf
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/config
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/desktop
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/guest-services
tmpfs                   796.0M    480.0K    795.5M   0% /host/run/host-services
cgroup_root              10.0M         0     10.0M   0% /host/sys/fs/cgroup
/dev/vda1               149.4G     34.8G    106.9G  25% /host/usr/libexec/kubernetes/kubelet-plugins
/dev/vda1               149.4G     34.8G    106.9G  25% /host/var/lib
/dev/vda1               149.4G     34.8G    106.9G  25% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker
tmpfs                     3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/tmp
overlay                   3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/rootfs
tmpfs                     3.9G      4.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/tmp
overlay                   3.9G      4.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/rootfs
tmpfs                     3.9G      8.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/tmp
overlay                   3.9G      8.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/rootfs
tmpfs                     3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/tmp
overlay                   3.9G    316.0K      3.9G   0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/rootfs
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/tmp
overlay                   3.9G         0      3.9G   0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/rootfs
overlay                 149.4G     34.8G    106.9G  25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
overlay                 149.4G     34.8G    106.9G  25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
tmpfs                    64.0M         0     64.0M   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev
shm                      64.0M         0     64.0M   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev/shm
tmpfs                     3.9G         0      3.9G   0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/sys/fs/cgroup
tmpfs                     3.9G         0      3.9G   0% /host/var/log
tmpfs                   796.0M    480.0K    795.5M   0% /host/var/run/linuxkit-containerd/containerd.sock
tmpfs                   796.0M    480.0K    795.5M   0% /host/var/run/linuxkit-external-logging.sock
grpcfuse                931.5G     14.3G    768.1G   2% /host/host_mnt
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/resolv.conf
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/hostname
/dev/vda1               149.4G     34.8G    106.9G  25% /etc/hosts
tmpfs                     3.9G         0      3.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.9G         0      3.9G   0% /sys/firmware

我以前从未见过这种情况,它是几周前才开始的。它不会一直发生,一切都可以正常工作数小时,有时甚至数天,然后崩溃。 None 从事同一项目的同事遇到了同样的问题。

Docker Mac 桌面:v3.6.0 (67351)
Docker 引擎:v20.10.8
撰写:v1.29.2
MacOS:v11.5.1 (20G80)

还有其他人遇到过这个问题吗?关于如何进一步调查问题有什么建议吗?

当我停用名为 Use gRPC FUSE for file sharing 的选项时,问题似乎消失了。我不知道这是否与我的磁盘或机器上的问题有关,但我很高兴它现在可以工作了。