Docker 容器的 sshfs 挂载冻结，但仅当由 Python 挂载时
Docker container's sshfs mount freezes, but only when mounted by Python
我有一台开发笔记本电脑 (Mint 19.3) 和一台测试服务器 (Ubuntu 18.04.4 LTS)。
笔记本电脑是Dockerversion 19.03.5, build 633a0ea838，服务器是Dockerversion 19.03.12, build 48a66213fe
我是运行 Python容器内的3.6代码，它使用subprocess（下面的代码）创建一个sshfs挂载到第三个服务器，之后python 代码遍历挂载目录。
在我的开发笔记本电脑上一切正常。但是在服务器上，目录挂载（并且可以通过 mount 命令看到）但是 cd 进入目录只是挂起 ，并且 Python 代码的后续 walk只是挂起。 （注意：python 代码永远不会崩溃或出错。它只会永远挂起。）
但是，如果我在容器的命令行手动使用相同的 sshfs 命令，目录工作正常。
我不知道如何解决这个问题。
===2020-09-25更新===
好的。由于 Python 代码使用子进程，sshfs 挂载显然可用于任何想要使用它的终端 windows。
我尝试从容器内的新终端 window 访问安装，但是当我 cd 到安装时 - window 只是冻结。
好吧，我把所有的东西都放在了一夜之间 - 现在当我尝试 cd 到挂载时......它起作用了。这就像坐骑必须坐上几个小时才能工作。
有什么想法吗？
Python代码
@target_dir.setter
def target_dir(self, value):
if value is None:
_tmp = tempfile.mkdtemp()
self._TARGETDIR = _tmp
else:
self._TARGETDIR = value
def mount(self):
# sshfs username@server-worker01:/test_project/ /mnt/test3 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = self.user,
S = self.server,
RD = self.remote_dir,
LD = self.target_dir)
command += " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
在容器内手动挂载有效
root@328100dd78be:/# mkdir /mnt/test
root@328100dd78be:/# sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=/etc/ssl/certs/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
root@328100dd78be:/# cd /mnt/test
root@328100dd78be:/mnt/test# ls -la
total 40
drwxr-s--- 1 16543 41500 210 Aug 6 07:07 .
drwxr-xr-x 1 root root 4096 Sep 24 14:27 ..
drwxr-s--- 1 16543 41500 187 Jul 27 11:16 configFiles
(...snip...)
drwx--S--- 1 16543 41500 104 May 6 14:03 submission
但是如果我让代码挂载它，目录就会挂起
root@328100dd78be:# ODCFFileCrawl.py
2020-09-24 14:31:16,722 - root - INFO - logger started! with:{'loglevel': '10'}
sshfs username@server-worker01:/test_project/ /tmp/tmp0wwjcg_6 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other,StrictHostKeyChecking=no
2020-09-24 14:31:16,797 - root - WARNING - Mounted remote filesystem 'server-worker01:/test_project/' to '/tmp/tmp7fig76gx'.
(change to new console)
root@328100dd78be:/# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/RWED5GN6GYRRGECDFN4LLIVVU7:/var/lib/docker/overlay2/l/7MTLBYSZ4ARQHKQ65TYBY5VSHW:/var/lib/docker/overlay2/l/EJX3L6YHRYB5IICXKAIH4PHT5I:/var/lib/docker/overlay2/l/AFJWYKSAAP6E7RMIU3H2PUMRSN:/var/lib/docker/overlay2/l/CU3AWZFBMNMQLQSBUQSVCHOKWC:/var/lib/docker/overlay2/l/RBJZT34EXSVZD3UYORTLM6L5UX:/var/lib/docker/overlay2/l/OJ7YFP5Z4SEETA2UVO35HQ2K53:/var/lib/docker/overlay2/l/CBU6F6UI47VHQ2BW5NMGJE64UA:/var/lib/docker/overlay2/l/62ZEX3XIFI7VYTLYYJX6XNLJTM:/var/lib/docker/overlay2/l/NBRXSW53NRLL5KRGYQ2BDL2ICZ:/var/lib/docker/overlay2/l/W5DJJWYP2VAYHTNDDHPY2KK2N3:/var/lib/docker/overlay2/l/ZOR2MOYW2NICOX3YVUUY6VXKVW:/var/lib/docker/overlay2/l/WLZEM73R55XKTPNVFCC5TRDCKT:/var/lib/docker/overlay2/l/PARGLNJTE4CM7PTTS5UOLXAIOC:/var/lib/docker/overlay2/l/JDA4NCWU5RN5YNVC63USV4O3VC:/var/lib/docker/overlay2/l/CEGTXJOQH65FTIZLAJNTAPF24R:/var/lib/docker/overlay2/l/B4VV7H463RL3THLES637O57DUO:/var/lib/docker/overlay2/l/KCENZYGKRWT7BRZC2HRUKSB4C5:/var/lib/docker/overlay2/l/EOYAIHOSSFBSX7X5HEWWZQGXS4:/var/lib/docker/overlay2/l/T2ZFALNJHY37UVCYHXD6GC36R5:/var/lib/docker/overlay2/l/4ACYIA6XBKYGFV5BUHYWTOALAY:/var/lib/docker/overlay2/l/PWZLNT3WAZPWPPYFOHZQ2SBHMG:/var/lib/docker/overlay2/l/W7UH56VKQDQH65GOWIGCYAMR3U,upperdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/diff,workdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/vda1 on /etc/ssl/certs type ext4 (rw,relatime,data=ordered)
/dev/vdb1 on /opt/vep/.vep type ext4 (rw,relatime,data=ordered)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
username@server-worker01:/test_project/ on /mnt/test type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
username@server-worker01:/test_project/ on /tmp/tmp7fig76gx type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
root@328100dd78be:# cd /tmp/tmp7fig76gx
(hangs forever)
===2020-12-04 @colidyre===
更新
试试这个...
mount.py
import tempfile
import subprocess
# target_dir = The local full path to which the remote directory will be mounted
target_dir = tempfile.mkdtemp()
user = "<Your username on the remote server >"
server = "<The server name for the remote server>"
remote_dir = "<Full path of the remote directory>"
key = "<Full path of the ssh private key for the remote server>"
options = " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = user,
S = server,
RD = remote_dir,
LD = target_dir)
command += options
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
Docker文件的代码
注意：这只是来自 https://hub.docker.com/r/ensemblorg/ensembl-vep/ 的工厂 VEP Docker 容器，中间添加了一些行以将 mount.py 命令复制到 /usr/bin，入口点更改为 [=25] =].请参阅部分...
##########################################
# Setup environment for mount.py
##########################################
Docker文件
###################################################
# Stage 1 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04 as builder
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
apt-utils \
python3 \
python3-pip \
git \
libpng-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
perl \
perl-base \
unzip \
wget
# wget && \
# rm -rf /var/lib/apt/lists/*
# Setup VEP environment
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV BRANCH release/101
# Working directory
WORKDIR $OPT_SRC
# Clone/download repositories/libraries
RUN if [ "$BRANCH" = "master" ]; \
then export BRANCH_OPT=""; \
else export BRANCH_OPT="-b $BRANCH"; \
fi && \
# Get ensembl cpanfile in order to get the list of the required Perl libraries
wget -q "https://raw.githubusercontent.com/Ensembl/ensembl/$BRANCH/cpanfile" -O "ensembl_cpanfile" && \
# Clone ensembl-vep git repository
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-vep.git && chmod u+x ensembl-vep/*.pl && \
# Clone ensembl-variation git repository and compile C code
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-variation.git && \
mkdir var_c_code && \
cp ensembl-variation/C_code/*.c ensembl-variation/C_code/Makefile var_c_code/ && \
rm -rf ensembl-variation && \
chmod u+x var_c_code/* && \
# Clone bioperl-ext git repository - used by Haplosaurus
git clone --depth 1 https://github.com/bioperl/bioperl-ext.git && \
# Download ensembl-xs - it contains compiled versions of certain key subroutines used in VEP
wget https://github.com/Ensembl/ensembl-xs/archive/2.3.2.zip -O ensembl-xs.zip && \
unzip -q ensembl-xs.zip && mv ensembl-xs-2.3.2 ensembl-xs && rm -rf ensembl-xs.zip && \
# Clone/Download other repositories: bioperl-live is needed so the cpanm dependencies installation from the ensembl-vep/cpanfile file takes less disk space
ensembl-vep/travisci/get_dependencies.sh && \
# Only keep the bioperl-live "Bio" library
mv bioperl-live bioperl-live_bak && mkdir bioperl-live && mv bioperl-live_bak/Bio bioperl-live/ && rm -rf bioperl-live_bak && \
## A lot of cleanup on the imported libraries, in order to reduce the docker image ##
rm -rf Bio-HTS/.??* Bio-HTS/Changes Bio-HTS/DISCLAIMER Bio-HTS/MANIFEST* Bio-HTS/README Bio-HTS/scripts Bio-HTS/t Bio-HTS/travisci \
bioperl-ext/.??* bioperl-ext/Bio/SeqIO bioperl-ext/Bio/Tools bioperl-ext/Makefile.PL bioperl-ext/README* bioperl-ext/t bioperl-ext/examples \
ensembl-vep/.??* ensembl-vep/docker \
ensembl-xs/.??* ensembl-xs/TODO ensembl-xs/Changes ensembl-xs/INSTALL ensembl-xs/MANIFEST ensembl-xs/README ensembl-xs/t ensembl-xs/travisci \
htslib/.??* htslib/INSTALL htslib/NEWS htslib/README* htslib/test && \
# Only keep needed kent-335_base libraries for VEP - used by Bio::DB::BigFile (bigWig parsing)
mv kent-335_base kent-335_base_bak && mkdir -p kent-335_base/src && \
cp -R kent-335_base_bak/src/lib kent-335_base_bak/src/inc kent-335_base_bak/src/jkOwnLib kent-335_base/src/ && \
cp kent-335_base_bak/src/*.sh kent-335_base/src/ && \
rm -rf kent-335_base_bak
# Setup bioperl-ext
WORKDIR bioperl-ext/Bio/Ext/Align/
RUN perl -pi -e"s|(cd libs.+)CFLAGS=\\'|$1CFLAGS=\\'-fPIC |" Makefile.PL
# Install htslib binaries (for 'bgzip' and 'tabix')
# htslib requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
WORKDIR $HTSLIB_DIR
RUN make install && rm -f Makefile *.c
# Compile Variation LD C scripts
WORKDIR $OPT_SRC/var_c_code
RUN make && rm -f Makefile *.c
###################################################
# Stage 2 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
python3 \
python3-pip \
git \
cpanminus \
curl \
libmysqlclient-dev \
libpng-dev \
libssl-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
locales \
openssl \
perl \
perl-base \
unzip \
sshfs \
vim && \
apt-get -y purge manpages-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# gitpython must be installed before setup.py is run
RUN pip3 install gitpython
##########################################
# Setup environment for mount.py
##########################################
# Takes the local FUSE conf as the base. This should be properly configured
# with "user_allow_other" enabled
COPY mount.py /usr/bin/mount.py
RUN chown root:root /usr/bin/mount.py
RUN chmod 755 /usr/bin/mount.py
# To hold the GIT downloads prior to install
RUN mkdir /src
RUN chmod 777 /src
##########################################
# Setup VEP environment
##########################################
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV PERL5LIB_TMP $PERL5LIB:$OPT_SRC/ensembl-vep:$OPT_SRC/ensembl-vep/modules
ENV PERL5LIB $PERL5LIB_TMP:$OPT_SRC/bioperl-live
ENV KENT_SRC $OPT/src/kent-335_base/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV MACHTYPE x86_64
ENV DEPS $OPT_SRC
ENV PATH $OPT_SRC/ensembl-vep:$OPT_SRC/var_c_code:$PATH
ENV LANG_VAR en_US.UTF-8
# Create vep user
RUN useradd -r -m -U -d "$OPT" -s /bin/bash -c "VEP User" -p '' vep && usermod -a -G sudo vep && mkdir -p $OPT_SRC
USER vep
# Copy downloaded libraries (stage 1) to this image (stage 2)
COPY --chown=vep:vep --from=builder $OPT_SRC $OPT_SRC
#############################################################
# Change user to root for the following complilations/installations
USER root
# Install bioperl-ext, faster alignments for haplo (XS-based BioPerl extensions to C libraries)
WORKDIR $OPT_SRC/bioperl-ext/Bio/Ext/Align/
RUN perl Makefile.PL && make && make install && rm -f Makefile*
# Install ensembl-xs, faster run using re-implementation in C of some of the Perl subroutines
WORKDIR $OPT_SRC/ensembl-xs
RUN perl Makefile.PL && make && make install && rm -f Makefile* cpanfile
WORKDIR $OPT_SRC
# Install/compile more libraries
RUN ensembl-vep/travisci/build_c.sh && \
# Remove unused Bio-DB-HTS files
rm -rf Bio-HTS/cpanfile Bio-HTS/Build.PL Bio-HTS/Build Bio-HTS/_build Bio-HTS/INSTALL.pl && \
# Install ensembl perl dependencies (cpanm)
cpanm --installdeps --with-recommends --notest --cpanfile ensembl_cpanfile . && \
cpanm --installdeps --with-recommends --notest --cpanfile ensembl-vep/cpanfile . && \
# Delete bioperl and cpanfiles after the cpanm installs as bioperl will be reinstalled by the INSTALL.pl script
rm -rf bioperl-live ensembl_cpanfile ensembl-vep/cpanfile && \
# Configure "locale", see https://github.com/rocker-org/rocker/issues/19
echo "$LANG_VAR UTF-8" >> /etc/locale.gen && locale-gen en_US.utf8 && \
/usr/sbin/update-locale LANG=$LANG_VAR && \
# Copy htslib executables. It also requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
cp $HTSLIB_DIR/bgzip $HTSLIB_DIR/tabix $HTSLIB_DIR/htsfile /usr/local/bin/
ENV LC_ALL $LANG_VAR
ENV LANG $LANG_VAR
# Switch back to vep user
USER vep
ENV PERL5LIB $PERL5LIB_TMP
# Final steps for VEP
WORKDIR $OPT_SRC/ensembl-vep
# Update bash profile
RUN echo >> $OPT/.profile && \
echo PATH=$PATH:$PATH >> $OPT/.profile && \
echo export PATH >> $OPT/.profile && \
# Run INSTALL.pl and remove the ensemb-vep tests and travis
./INSTALL.pl -a a -l -n && rm -rf t travisci .travis.yml
ENTRYPOINT ["mount.py"]
# CMD tail -f /dev/null
我假设您想使用 SSHFS 将某个服务器的目录挂载到容器的文件系统。您可以将该指令添加到 Dockerfile：
FROM ubuntu:18.04
RUN apt-get update && apt-get -y install stuff
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
您可以在此下方添加 Dockerfile 中的其他内容。您应该将 sshprivkey 文件添加到当前目录。
现在如果你构建它：
docker build -t your_desired_image_name .
然后 运行 使用该图像的容器并检查它是否有效：
docker run --privileged -it your_desired_image_name
root@container_id$ ls /mnt/test
请记住添加 --privileged 标志以使其正常工作。我从 here.
中发现的
服务器 Directory/Directories 可以使用 SSHFS 实用程序直接挂载到容器的文件系统。
Docker 文件
FROM ubuntu:18.04
RUN apt-get update
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
容器是使用 ubuntu:18.04 基础映像创建的。
FROM ubuntu:18.04
更新包列表。 此命令将根据基础映像进行更改。可以更新这里命令以安装所需的包
RUN apt-get update
已复制 sshprivkey 与 sshfs 一起使用以获得将服务器直接挂载到容器的权限
COPY sshprivkey .ssh/sshprivkey
创建该测试目录的服务器目录应安装在何处
RUN mkdir /mnt/test
运行 sshfs 使用 sshprivkey 命令在先前创建的测试目录中挂载服务器目录
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
要从当前目录中的 Dockerfile 构建映像，请使用以下命令：
docker build -t your_desired_image_name .
运行ning 交互式会话容器来自先前创建的图像，遵循它使用的命令：
docker run --cap-add SYS_ADMIN -it your_desired_image_name
容器交互会话启动后，我们可以检查它是否有效：
root@container_id$ ls /mnt/test
这里我使用了--cap-add给容器挂载目录的权限
注意： 最好使用 --cap-add 根据需要添加有限权限，而不是使用 --privileged 直到没有其他出于安全目的的工作是给那个容器太多的权限不好
您可以阅读有关 docker 权限的更多信息 here
我有一台开发笔记本电脑 (Mint 19.3) 和一台测试服务器 (Ubuntu 18.04.4 LTS)。
笔记本电脑是Dockerversion 19.03.5, build 633a0ea838，服务器是Dockerversion 19.03.12, build 48a66213fe
我是运行 Python容器内的3.6代码，它使用subprocess（下面的代码）创建一个sshfs挂载到第三个服务器，之后python 代码遍历挂载目录。
在我的开发笔记本电脑上一切正常。但是在服务器上，目录挂载（并且可以通过 mount 命令看到）但是 cd 进入目录只是挂起 ，并且 Python 代码的后续 walk只是挂起。 （注意：python 代码永远不会崩溃或出错。它只会永远挂起。）
但是，如果我在容器的命令行手动使用相同的 sshfs 命令，目录工作正常。
我不知道如何解决这个问题。
===2020-09-25更新===
好的。由于 Python 代码使用子进程，sshfs 挂载显然可用于任何想要使用它的终端 windows。
我尝试从容器内的新终端 window 访问安装，但是当我 cd 到安装时 - window 只是冻结。
好吧，我把所有的东西都放在了一夜之间 - 现在当我尝试 cd 到挂载时......它起作用了。这就像坐骑必须坐上几个小时才能工作。
有什么想法吗？
Python代码
@target_dir.setter
def target_dir(self, value):
if value is None:
_tmp = tempfile.mkdtemp()
self._TARGETDIR = _tmp
else:
self._TARGETDIR = value
def mount(self):
# sshfs username@server-worker01:/test_project/ /mnt/test3 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = self.user,
S = self.server,
RD = self.remote_dir,
LD = self.target_dir)
command += " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
在容器内手动挂载有效
root@328100dd78be:/# mkdir /mnt/test
root@328100dd78be:/# sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=/etc/ssl/certs/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
root@328100dd78be:/# cd /mnt/test
root@328100dd78be:/mnt/test# ls -la
total 40
drwxr-s--- 1 16543 41500 210 Aug 6 07:07 .
drwxr-xr-x 1 root root 4096 Sep 24 14:27 ..
drwxr-s--- 1 16543 41500 187 Jul 27 11:16 configFiles
(...snip...)
drwx--S--- 1 16543 41500 104 May 6 14:03 submission
但是如果我让代码挂载它，目录就会挂起
root@328100dd78be:# ODCFFileCrawl.py
2020-09-24 14:31:16,722 - root - INFO - logger started! with:{'loglevel': '10'}
sshfs username@server-worker01:/test_project/ /tmp/tmp0wwjcg_6 -o IdentityFile=/home/username/.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other,StrictHostKeyChecking=no
2020-09-24 14:31:16,797 - root - WARNING - Mounted remote filesystem 'server-worker01:/test_project/' to '/tmp/tmp7fig76gx'.
(change to new console)
root@328100dd78be:/# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/RWED5GN6GYRRGECDFN4LLIVVU7:/var/lib/docker/overlay2/l/7MTLBYSZ4ARQHKQ65TYBY5VSHW:/var/lib/docker/overlay2/l/EJX3L6YHRYB5IICXKAIH4PHT5I:/var/lib/docker/overlay2/l/AFJWYKSAAP6E7RMIU3H2PUMRSN:/var/lib/docker/overlay2/l/CU3AWZFBMNMQLQSBUQSVCHOKWC:/var/lib/docker/overlay2/l/RBJZT34EXSVZD3UYORTLM6L5UX:/var/lib/docker/overlay2/l/OJ7YFP5Z4SEETA2UVO35HQ2K53:/var/lib/docker/overlay2/l/CBU6F6UI47VHQ2BW5NMGJE64UA:/var/lib/docker/overlay2/l/62ZEX3XIFI7VYTLYYJX6XNLJTM:/var/lib/docker/overlay2/l/NBRXSW53NRLL5KRGYQ2BDL2ICZ:/var/lib/docker/overlay2/l/W5DJJWYP2VAYHTNDDHPY2KK2N3:/var/lib/docker/overlay2/l/ZOR2MOYW2NICOX3YVUUY6VXKVW:/var/lib/docker/overlay2/l/WLZEM73R55XKTPNVFCC5TRDCKT:/var/lib/docker/overlay2/l/PARGLNJTE4CM7PTTS5UOLXAIOC:/var/lib/docker/overlay2/l/JDA4NCWU5RN5YNVC63USV4O3VC:/var/lib/docker/overlay2/l/CEGTXJOQH65FTIZLAJNTAPF24R:/var/lib/docker/overlay2/l/B4VV7H463RL3THLES637O57DUO:/var/lib/docker/overlay2/l/KCENZYGKRWT7BRZC2HRUKSB4C5:/var/lib/docker/overlay2/l/EOYAIHOSSFBSX7X5HEWWZQGXS4:/var/lib/docker/overlay2/l/T2ZFALNJHY37UVCYHXD6GC36R5:/var/lib/docker/overlay2/l/4ACYIA6XBKYGFV5BUHYWTOALAY:/var/lib/docker/overlay2/l/PWZLNT3WAZPWPPYFOHZQ2SBHMG:/var/lib/docker/overlay2/l/W7UH56VKQDQH65GOWIGCYAMR3U,upperdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/diff,workdir=/var/lib/docker/overlay2/9c2385aa1221d4fbcac321bfb7862dae23677bdf594ddea803315144b5124617/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/vda1 on /etc/ssl/certs type ext4 (rw,relatime,data=ordered)
/dev/vdb1 on /opt/vep/.vep type ext4 (rw,relatime,data=ordered)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
username@server-worker01:/test_project/ on /mnt/test type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
username@server-worker01:/test_project/ on /tmp/tmp7fig76gx type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
root@328100dd78be:# cd /tmp/tmp7fig76gx
(hangs forever)
===2020-12-04 @colidyre===
更新试试这个...
mount.py
import tempfile
import subprocess
# target_dir = The local full path to which the remote directory will be mounted
target_dir = tempfile.mkdtemp()
user = "<Your username on the remote server >"
server = "<The server name for the remote server>"
remote_dir = "<Full path of the remote directory>"
key = "<Full path of the ssh private key for the remote server>"
options = " -o IdentityFile={IDF},auto_cache,reconnect,transform_symlinks,follow_symlinks,allow_other".format(IDF = key)
command = "sshfs {U}@{S}:{RD} {LD}".format(
U = user,
S = server,
RD = remote_dir,
LD = target_dir)
command += options
cp = subprocess.getoutput(command)
if cp != "":
err = "Something appears to be wrong with the mount. ({})".format(cp)
raise RuntimeError(err)
Docker文件的代码
注意：这只是来自 https://hub.docker.com/r/ensemblorg/ensembl-vep/ 的工厂 VEP Docker 容器，中间添加了一些行以将 mount.py 命令复制到 /usr/bin，入口点更改为 [=25] =].请参阅部分...
##########################################
# Setup environment for mount.py
##########################################
Docker文件
###################################################
# Stage 1 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04 as builder
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
apt-utils \
python3 \
python3-pip \
git \
libpng-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
perl \
perl-base \
unzip \
wget
# wget && \
# rm -rf /var/lib/apt/lists/*
# Setup VEP environment
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV BRANCH release/101
# Working directory
WORKDIR $OPT_SRC
# Clone/download repositories/libraries
RUN if [ "$BRANCH" = "master" ]; \
then export BRANCH_OPT=""; \
else export BRANCH_OPT="-b $BRANCH"; \
fi && \
# Get ensembl cpanfile in order to get the list of the required Perl libraries
wget -q "https://raw.githubusercontent.com/Ensembl/ensembl/$BRANCH/cpanfile" -O "ensembl_cpanfile" && \
# Clone ensembl-vep git repository
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-vep.git && chmod u+x ensembl-vep/*.pl && \
# Clone ensembl-variation git repository and compile C code
git clone $BRANCH_OPT --depth 1 https://github.com/Ensembl/ensembl-variation.git && \
mkdir var_c_code && \
cp ensembl-variation/C_code/*.c ensembl-variation/C_code/Makefile var_c_code/ && \
rm -rf ensembl-variation && \
chmod u+x var_c_code/* && \
# Clone bioperl-ext git repository - used by Haplosaurus
git clone --depth 1 https://github.com/bioperl/bioperl-ext.git && \
# Download ensembl-xs - it contains compiled versions of certain key subroutines used in VEP
wget https://github.com/Ensembl/ensembl-xs/archive/2.3.2.zip -O ensembl-xs.zip && \
unzip -q ensembl-xs.zip && mv ensembl-xs-2.3.2 ensembl-xs && rm -rf ensembl-xs.zip && \
# Clone/Download other repositories: bioperl-live is needed so the cpanm dependencies installation from the ensembl-vep/cpanfile file takes less disk space
ensembl-vep/travisci/get_dependencies.sh && \
# Only keep the bioperl-live "Bio" library
mv bioperl-live bioperl-live_bak && mkdir bioperl-live && mv bioperl-live_bak/Bio bioperl-live/ && rm -rf bioperl-live_bak && \
## A lot of cleanup on the imported libraries, in order to reduce the docker image ##
rm -rf Bio-HTS/.??* Bio-HTS/Changes Bio-HTS/DISCLAIMER Bio-HTS/MANIFEST* Bio-HTS/README Bio-HTS/scripts Bio-HTS/t Bio-HTS/travisci \
bioperl-ext/.??* bioperl-ext/Bio/SeqIO bioperl-ext/Bio/Tools bioperl-ext/Makefile.PL bioperl-ext/README* bioperl-ext/t bioperl-ext/examples \
ensembl-vep/.??* ensembl-vep/docker \
ensembl-xs/.??* ensembl-xs/TODO ensembl-xs/Changes ensembl-xs/INSTALL ensembl-xs/MANIFEST ensembl-xs/README ensembl-xs/t ensembl-xs/travisci \
htslib/.??* htslib/INSTALL htslib/NEWS htslib/README* htslib/test && \
# Only keep needed kent-335_base libraries for VEP - used by Bio::DB::BigFile (bigWig parsing)
mv kent-335_base kent-335_base_bak && mkdir -p kent-335_base/src && \
cp -R kent-335_base_bak/src/lib kent-335_base_bak/src/inc kent-335_base_bak/src/jkOwnLib kent-335_base/src/ && \
cp kent-335_base_bak/src/*.sh kent-335_base/src/ && \
rm -rf kent-335_base_bak
# Setup bioperl-ext
WORKDIR bioperl-ext/Bio/Ext/Align/
RUN perl -pi -e"s|(cd libs.+)CFLAGS=\\'|$1CFLAGS=\\'-fPIC |" Makefile.PL
# Install htslib binaries (for 'bgzip' and 'tabix')
# htslib requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
WORKDIR $HTSLIB_DIR
RUN make install && rm -f Makefile *.c
# Compile Variation LD C scripts
WORKDIR $OPT_SRC/var_c_code
RUN make && rm -f Makefile *.c
###################################################
# Stage 2 - docker container to build ensembl-vep #
###################################################
FROM ubuntu:18.04
# Update aptitude and install some required packages
# a lot of them are required for Bio::DB::BigFile
RUN apt-get update && apt-get -y install \
build-essential \
python3 \
python3-pip \
git \
cpanminus \
curl \
libmysqlclient-dev \
libpng-dev \
libssl-dev \
zlib1g-dev \
libbz2-dev \
liblzma-dev \
locales \
openssl \
perl \
perl-base \
unzip \
sshfs \
vim && \
apt-get -y purge manpages-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# gitpython must be installed before setup.py is run
RUN pip3 install gitpython
##########################################
# Setup environment for mount.py
##########################################
# Takes the local FUSE conf as the base. This should be properly configured
# with "user_allow_other" enabled
COPY mount.py /usr/bin/mount.py
RUN chown root:root /usr/bin/mount.py
RUN chmod 755 /usr/bin/mount.py
# To hold the GIT downloads prior to install
RUN mkdir /src
RUN chmod 777 /src
##########################################
# Setup VEP environment
##########################################
ENV OPT /opt/vep
ENV OPT_SRC $OPT/src
ENV PERL5LIB_TMP $PERL5LIB:$OPT_SRC/ensembl-vep:$OPT_SRC/ensembl-vep/modules
ENV PERL5LIB $PERL5LIB_TMP:$OPT_SRC/bioperl-live
ENV KENT_SRC $OPT/src/kent-335_base/src
ENV HTSLIB_DIR $OPT_SRC/htslib
ENV MACHTYPE x86_64
ENV DEPS $OPT_SRC
ENV PATH $OPT_SRC/ensembl-vep:$OPT_SRC/var_c_code:$PATH
ENV LANG_VAR en_US.UTF-8
# Create vep user
RUN useradd -r -m -U -d "$OPT" -s /bin/bash -c "VEP User" -p '' vep && usermod -a -G sudo vep && mkdir -p $OPT_SRC
USER vep
# Copy downloaded libraries (stage 1) to this image (stage 2)
COPY --chown=vep:vep --from=builder $OPT_SRC $OPT_SRC
#############################################################
# Change user to root for the following complilations/installations
USER root
# Install bioperl-ext, faster alignments for haplo (XS-based BioPerl extensions to C libraries)
WORKDIR $OPT_SRC/bioperl-ext/Bio/Ext/Align/
RUN perl Makefile.PL && make && make install && rm -f Makefile*
# Install ensembl-xs, faster run using re-implementation in C of some of the Perl subroutines
WORKDIR $OPT_SRC/ensembl-xs
RUN perl Makefile.PL && make && make install && rm -f Makefile* cpanfile
WORKDIR $OPT_SRC
# Install/compile more libraries
RUN ensembl-vep/travisci/build_c.sh && \
# Remove unused Bio-DB-HTS files
rm -rf Bio-HTS/cpanfile Bio-HTS/Build.PL Bio-HTS/Build Bio-HTS/_build Bio-HTS/INSTALL.pl && \
# Install ensembl perl dependencies (cpanm)
cpanm --installdeps --with-recommends --notest --cpanfile ensembl_cpanfile . && \
cpanm --installdeps --with-recommends --notest --cpanfile ensembl-vep/cpanfile . && \
# Delete bioperl and cpanfiles after the cpanm installs as bioperl will be reinstalled by the INSTALL.pl script
rm -rf bioperl-live ensembl_cpanfile ensembl-vep/cpanfile && \
# Configure "locale", see https://github.com/rocker-org/rocker/issues/19
echo "$LANG_VAR UTF-8" >> /etc/locale.gen && locale-gen en_US.utf8 && \
/usr/sbin/update-locale LANG=$LANG_VAR && \
# Copy htslib executables. It also requires the packages 'zlib1g-dev', 'libbz2-dev' and 'liblzma-dev'
cp $HTSLIB_DIR/bgzip $HTSLIB_DIR/tabix $HTSLIB_DIR/htsfile /usr/local/bin/
ENV LC_ALL $LANG_VAR
ENV LANG $LANG_VAR
# Switch back to vep user
USER vep
ENV PERL5LIB $PERL5LIB_TMP
# Final steps for VEP
WORKDIR $OPT_SRC/ensembl-vep
# Update bash profile
RUN echo >> $OPT/.profile && \
echo PATH=$PATH:$PATH >> $OPT/.profile && \
echo export PATH >> $OPT/.profile && \
# Run INSTALL.pl and remove the ensemb-vep tests and travis
./INSTALL.pl -a a -l -n && rm -rf t travisci .travis.yml
ENTRYPOINT ["mount.py"]
# CMD tail -f /dev/null
我假设您想使用 SSHFS 将某个服务器的目录挂载到容器的文件系统。您可以将该指令添加到 Dockerfile：
FROM ubuntu:18.04
RUN apt-get update && apt-get -y install stuff
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
您可以在此下方添加 Dockerfile 中的其他内容。您应该将 sshprivkey 文件添加到当前目录。
现在如果你构建它：
docker build -t your_desired_image_name .
然后 运行 使用该图像的容器并检查它是否有效：
docker run --privileged -it your_desired_image_name
root@container_id$ ls /mnt/test
请记住添加 --privileged 标志以使其正常工作。我从 here.
服务器 Directory/Directories 可以使用 SSHFS 实用程序直接挂载到容器的文件系统。
Docker 文件
FROM ubuntu:18.04
RUN apt-get update
COPY sshprivkey .ssh/sshprivkey
RUN mkdir /mnt/test
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
容器是使用 ubuntu:18.04 基础映像创建的。
FROM ubuntu:18.04
更新包列表。 此命令将根据基础映像进行更改。可以更新这里命令以安装所需的包
RUN apt-get update
已复制 sshprivkey 与 sshfs 一起使用以获得将服务器直接挂载到容器的权限
COPY sshprivkey .ssh/sshprivkey
创建该测试目录的服务器目录应安装在何处
RUN mkdir /mnt/test
运行 sshfs 使用 sshprivkey 命令在先前创建的测试目录中挂载服务器目录
RUN sshfs username@server-worker01:/test_project/ /mnt/test -o IdentityFile=.ssh/sshprivkey,auto_cache,reconnect,transform_symlinks,follow_symlinks
要从当前目录中的 Dockerfile 构建映像，请使用以下命令：
docker build -t your_desired_image_name .
运行ning 交互式会话容器来自先前创建的图像，遵循它使用的命令：
docker run --cap-add SYS_ADMIN -it your_desired_image_name
容器交互会话启动后，我们可以检查它是否有效：
root@container_id$ ls /mnt/test
这里我使用了--cap-add给容器挂载目录的权限
注意： 最好使用 --cap-add 根据需要添加有限权限，而不是使用 --privileged 直到没有其他出于安全目的的工作是给那个容器太多的权限不好
您可以阅读有关 docker 权限的更多信息 here