Gitlab - 获取作业时发生错误
Gitlab - An error occurred while fetching the job
我是 运行 最新版本的 Gitlab 社区版，位于 jwilder nginx 代理后面的 Docker 容器中。机架攻击 Gem 被配置禁用，但是当 Gitlab Runner 作业无法针对注册表（位于同一域但端口 4567）进行身份验证时，Gitlab 中会出现红色横幅：
An error occurred while fetching the job
当我刷新 Gitlab 实例时，我只看到 403 HTTP 状态，它告诉我，我不再被允许访问 Gitlab。即使我在另一个浏览器 window 中使用不同的用户登录（即使是私人 window）。
我不确定这里发生了什么，但愿意分享尽可能多的信息。但我真的不知道从哪里开始，因为唯一有意义的是禁用的 Rack Attack 插件。
我试图通过丢弃当前容器来构建新版本的 docker 容器。我还清除了绑定的 redis 实例，以确保机架攻击对当前行为确实没有影响。
docker 信息：
Containers: 58
Running: 21
Paused: 0
Stopped: 37
Images: 615
Server Version: 18.09.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1313
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: ...
Is Manager: true
ClusterID: ...
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: ...
Manager Addresses:
...:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.28GiB
Name: myserver.example.com
ID: JB7Q:3MNW:GJ6Q:...:EI5K:NKU3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
location=cloud
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
docker-compose.yml
version: '3'
services:
gitlab:
image: 'gitlab/gitlab-ce:latest'
container_name: gitlab
environment:
- VIRTUAL_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- VIRTUAL_PORT=80
- LETSENCRYPT_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}
- GITLAB_REGISTRY_ENABLED=true
- GITLAB_REGISTRY_HOST=${REPOSITORY_HOST}:4567
- GITLAB_REGISTRY_CERT_PATH=/certs/registry.example.com.chain.pem
- GITLAB_REGISTRY_KEY_PATH=/certs/registry.example.com.key
- REDIS_HOST=redis
- REDIS_PORT=6379
expose:
- 80
- 22
ports:
- '2222:22'
- '4567:4567'
restart: always
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/www/nginx-proxy/certs:/certs:ro'
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
redis:
restart: always
image: sameersbn/redis:latest
command:
- --loglevel warning
volumes:
- /srv/gitlab/redis:/var/lib/redis:Z
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
networks:
default:
external:
name: nginx-proxy
gitlab.rb配置：
external_url 'https://repository.example.com'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@repository.example.com'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@repository.example.com'
gitlab_rails['lfs_enabled'] = true
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = false
gitlab_rails['omniauth_block_auto_created_users'] = true
gitlab_rails['redis_host'] = "redis"
gitlab_rails['redis_port'] = 6379
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "changeme"
gitlab_rails['smtp_password'] = "changeme"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
registry_external_url 'https://repository.example.com:4567'
gitlab_rails['registry_enabled'] = true
registry['enable'] = true
redis['enable'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 4567
registry_nginx['listen_https'] = true
registry_nginx['ssl_certificate'] = "/certs/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/certs/registry.example.com.key"
registry_nginx['listen_addresses'] = ['*', '[::]']
registry_nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
.gitlab-ci.yml
image: php:7.3-fpm-alpine
cache:
paths:
- vendor/
- node_modules/
stages:
- build
- test
- release
variables:
DOCKER_DRIVER: overlay
NIGHTLY_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME
LATEST_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_BUILD_REF_NAME:latest
before_script:
- sh .gitlab/docker_install.sh > /dev/null
- docker info
- cp .env.example .env
- sed -i "s/CACHE_DRIVER=.*/CACHE_DRIVER=array/" .env
- sed -i "s/SESSION_DRIVER=.*/SESSION_DRIVER=array/" .env
- php composer.phar install
- php artisan key:generate
- npm install
- npm run prod
build:
stage: build
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker build --pull -t $NIGHTLY_IMAGE .
- docker build -t $NIGHTLY_IMAGE -f ./docker/php/Dockerfile.production --pull .
- docker push $NIGHTLY_IMAGE
- docker push $LATEST_IMAGE
test:7.2:
stage: test
image: php:7.2-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
test:7.3:
stage: test
image: php:7.3-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
release:
stage: release
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
.gitlab/docker_install.sh
#!/bin/sh
[[ ! -e /.dockerenv ]] && exit 0
set -xe
apk add --update \
docker \
libpq \
libpng-dev \
jpeg-dev \
freetype-dev \
libxrender \
fontconfig \
libxext-dev \
openldap-dev \
git \
nodejs nodejs-npm \
libzip-dev \
zip \
curl \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
&& docker-php-ext-configure zip --with-libzip \
&& docker-php-ext-install -j$(nproc) pdo_mysql mysqli gd zip ldap pcntl
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php
php -r "unlink('composer-setup.php');"
curl --location --output /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
所以目前我无法使用任何用户登录 Gitlab。必须可以再次登录，我想了解这里发生了什么以及为什么 Gitlab 会用禁止响应阻止每个 login/request。
看来是 Redis 的问题。一些帖子提到了机架攻击禁令，我很确定情况并非如此，因为默认情况下它是禁用的，但是在删除刷新所有 Redis 密钥后，我能够再次使用 Gitlab 而没有出现 403 Forbidden 错误。
https://gitlab.com/gitlab-org/gitlab-ce/issues/37090
所以我在我的案例中做了以下事情：
$ docker exec -it redis bash
$ redis-cli
$ FLUSHALL
我是 运行 最新版本的 Gitlab 社区版，位于 jwilder nginx 代理后面的 Docker 容器中。机架攻击 Gem 被配置禁用，但是当 Gitlab Runner 作业无法针对注册表（位于同一域但端口 4567）进行身份验证时，Gitlab 中会出现红色横幅：
An error occurred while fetching the job
当我刷新 Gitlab 实例时，我只看到 403 HTTP 状态，它告诉我，我不再被允许访问 Gitlab。即使我在另一个浏览器 window 中使用不同的用户登录（即使是私人 window）。
我不确定这里发生了什么，但愿意分享尽可能多的信息。但我真的不知道从哪里开始，因为唯一有意义的是禁用的 Rack Attack 插件。
我试图通过丢弃当前容器来构建新版本的 docker 容器。我还清除了绑定的 redis 实例，以确保机架攻击对当前行为确实没有影响。
docker 信息：
Containers: 58
Running: 21
Paused: 0
Stopped: 37
Images: 615
Server Version: 18.09.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1313
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: ...
Is Manager: true
ClusterID: ...
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: ...
Manager Addresses:
...:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.28GiB
Name: myserver.example.com
ID: JB7Q:3MNW:GJ6Q:...:EI5K:NKU3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
location=cloud
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
docker-compose.yml
version: '3'
services:
gitlab:
image: 'gitlab/gitlab-ce:latest'
container_name: gitlab
environment:
- VIRTUAL_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- VIRTUAL_PORT=80
- LETSENCRYPT_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}
- GITLAB_REGISTRY_ENABLED=true
- GITLAB_REGISTRY_HOST=${REPOSITORY_HOST}:4567
- GITLAB_REGISTRY_CERT_PATH=/certs/registry.example.com.chain.pem
- GITLAB_REGISTRY_KEY_PATH=/certs/registry.example.com.key
- REDIS_HOST=redis
- REDIS_PORT=6379
expose:
- 80
- 22
ports:
- '2222:22'
- '4567:4567'
restart: always
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/www/nginx-proxy/certs:/certs:ro'
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
redis:
restart: always
image: sameersbn/redis:latest
command:
- --loglevel warning
volumes:
- /srv/gitlab/redis:/var/lib/redis:Z
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
networks:
default:
external:
name: nginx-proxy
gitlab.rb配置：
external_url 'https://repository.example.com'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@repository.example.com'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@repository.example.com'
gitlab_rails['lfs_enabled'] = true
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = false
gitlab_rails['omniauth_block_auto_created_users'] = true
gitlab_rails['redis_host'] = "redis"
gitlab_rails['redis_port'] = 6379
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "changeme"
gitlab_rails['smtp_password'] = "changeme"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
registry_external_url 'https://repository.example.com:4567'
gitlab_rails['registry_enabled'] = true
registry['enable'] = true
redis['enable'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 4567
registry_nginx['listen_https'] = true
registry_nginx['ssl_certificate'] = "/certs/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/certs/registry.example.com.key"
registry_nginx['listen_addresses'] = ['*', '[::]']
registry_nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
.gitlab-ci.yml
image: php:7.3-fpm-alpine
cache:
paths:
- vendor/
- node_modules/
stages:
- build
- test
- release
variables:
DOCKER_DRIVER: overlay
NIGHTLY_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME
LATEST_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_BUILD_REF_NAME:latest
before_script:
- sh .gitlab/docker_install.sh > /dev/null
- docker info
- cp .env.example .env
- sed -i "s/CACHE_DRIVER=.*/CACHE_DRIVER=array/" .env
- sed -i "s/SESSION_DRIVER=.*/SESSION_DRIVER=array/" .env
- php composer.phar install
- php artisan key:generate
- npm install
- npm run prod
build:
stage: build
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker build --pull -t $NIGHTLY_IMAGE .
- docker build -t $NIGHTLY_IMAGE -f ./docker/php/Dockerfile.production --pull .
- docker push $NIGHTLY_IMAGE
- docker push $LATEST_IMAGE
test:7.2:
stage: test
image: php:7.2-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
test:7.3:
stage: test
image: php:7.3-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
release:
stage: release
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
.gitlab/docker_install.sh
#!/bin/sh
[[ ! -e /.dockerenv ]] && exit 0
set -xe
apk add --update \
docker \
libpq \
libpng-dev \
jpeg-dev \
freetype-dev \
libxrender \
fontconfig \
libxext-dev \
openldap-dev \
git \
nodejs nodejs-npm \
libzip-dev \
zip \
curl \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
&& docker-php-ext-configure zip --with-libzip \
&& docker-php-ext-install -j$(nproc) pdo_mysql mysqli gd zip ldap pcntl
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php
php -r "unlink('composer-setup.php');"
curl --location --output /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
所以目前我无法使用任何用户登录 Gitlab。必须可以再次登录，我想了解这里发生了什么以及为什么 Gitlab 会用禁止响应阻止每个 login/request。
看来是 Redis 的问题。一些帖子提到了机架攻击禁令，我很确定情况并非如此，因为默认情况下它是禁用的，但是在删除刷新所有 Redis 密钥后，我能够再次使用 Gitlab 而没有出现 403 Forbidden 错误。
https://gitlab.com/gitlab-org/gitlab-ce/issues/37090
所以我在我的案例中做了以下事情：
$ docker exec -it redis bash
$ redis-cli
$ FLUSHALL