来自共享 Gitlab 运行器的 SSH 停止工作

Question

这个以前工作过！

我在管道中的部署步骤 SSH 到 DO 框并从 docker 注册表中提取代码。如前所述，这以前有效，这是我当时 .gitlab-ci.yml 中的 deploy 步骤，它很好地启发了 Using SSH 下的 here:

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -S)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

有一次我尝试运行 deploy 步骤失败了。返回此错误：

...
 $ mkdir -p ~/.ssh
 $ echo "${DEPLOYMENT_SERVER_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
 $ chmod 600 ~/.ssh/id_rsa
 $ eval "$(ssh-agent -s)"
 Agent pid 22
 $ ssh-add ~/.ssh/id_rsa
 Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh gitlab@${DEPLOYMENT_SERVER_IP} "docker stop ${CI_PROJECT_NAME}; docker rm ${CI_PROJECT_NAME}; docker container prune -f; docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH}; docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}; docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest; docker run -d -p ${PORT}:${PORT} --restart always -m 2g --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
Running after_script
00:02
Uploading artifacts for failed job
00:01
 ERROR: Job failed: exit code 255

我最初设置的步骤

运行 ssh-keygen -t rsa -b 2048 在 DO 框 （无密码）
在 DO 框 authorized_keys 中添加了 public 键
将 private 密钥复制到 CI 变量中 DEPLOYMENT_SERVER_PRIVATE_KEY

我知道该端口已为 SSH 打开，因为我可以从我的本地计算机通过 SSH 连接到 gitlab 用户。我现在已将我的部署步骤 （基于 here, this article, & this one 的评论） 更改为：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - echo ${CI_REGISTRY_USER}
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

还是没用！ ssh 的详细日志记录吐出：

...
 $ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
 /usr/bin/ssh-agent
 $ eval $(ssh-agent -s)
 Agent pid 18
 $ echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
 Identity added: (stdin) ((stdin))
 $ mkdir -p ~/.ssh
 $ chmod 700 ~/.ssh
 $ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
 $ cat ~/.ssh/config
 Host *
    StrictHostKeyChecking no
 $ echo ${CI_REGISTRY_USER}
 gitlab-ci-token
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
 OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
 debug1: Reading configuration data /root/.ssh/config
 debug1: /root/.ssh/config line 1: Applying options for *
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug2: resolve_canonicalize: hostname 134.xxx.xxx.xxx is address
 Pseudo-terminal will not be allocated because stdin is not a terminal.
 debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
 debug2: ssh_connect_direct
 debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
 debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
 ERROR: Job failed: exit code 255

我还添加了 -T 选项 suggested here 来禁用伪 tty 分配，但所做的只是从日志中删除伪行。

编辑

查看 DO 框上的日志 (/var/log/auth.log)，我得到了错误：

Jun 22 15:53:37 exchange-apis sshd[16159]: Connection closed by 35.190.162.232 port 49750 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16160]: Connection closed by 35.190.162.232 port 49754 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16162]: Connection closed by 35.190.162.232 port 49752 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16163]: Unable to negotiate with 35.190.162.232 port 49756: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
Jun 22 15:53:38 exchange-apis sshd[16161]: Unable to negotiate with 35.190.162.232 port 49758: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]

谷歌搜索此错误，常见原因似乎是 OpenSSH 放弃了对 DSA 密钥的支持。但是，不确定为什么这会影响我，因为我生成了一个 RSA 密钥对。反正运行宁dpkg --list | grep openssh吐槽一下：

ii  openssh-client                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                    1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines

&sshd -v吐槽：

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

尽管如此，答案仍然有效； & here 所以我的 deploy 舞台现在是：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\tHostkeyAlgorithms +ssh-dss\n\tPubkeyAcceptedKeyTypes +ssh-dss\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - ssh -oHostKeyAlgorithms=+ssh-dss gitlab@${DEPLOYMENT_SERVER_IP} ls

仍然看不到它，我在运行ner 的输出和 DO 框中的日志中得到了同样的错误。有什么想法吗？

Answer 1

理想情况下，如果您可以登录到 DO 框，您将停止 ssh 服务，并启动 /usr/bin/sshd -de，以便在 SSH 守护程序端建立调试会话，并将日志写入 stderr （而不是系统消息）

但如果你不能，至少尝试生成一个没有密码的 rsa 密钥，用于测试。这意味着您不需要 ssh-agent。
并尝试 ssh -Tv gitlab@${DEPLOYMENT_SERVER_IP} ls 查看那里生成的日志。

试试 classic PEM format

ssh-keygen -t rsa -P "" -m PEM

after editing the pipeline a bit more, I've noticed that it is actually this line that is causing the issue: ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts

如果它导致格式错误 ~/.ssh/known_hosts，可能会出现这种情况，尤其是 ${DEPLOYMENT_SERVER_IP} 设置不正确时。
尝试将 echo "DEPLOYMENT_SERVER_IP='${DEPLOYMENT_SERVER_IP}'" 和 cat ~/.ssh/known_hosts 命令添加到 before_script 部分，以了解更多信息。

来自共享 Gitlab 运行器的 SSH 停止工作

SSH from shared Gitlab runner stopped working

ssh

openssh

gitlab

gitlab-ci

gitlab-ci-runner