Gitlab CI：opencv-python 构建轮失败

Question

我正在为我在 gitlab 中的 python/django 项目开发 CI/CD。

我有一个错误 -- Gitlab CI: opencv-python

构建轮失败

完整的 gitlab ci 日志 -- https://pastebin.com/pZdZ6ws2

我在 build_pip 阶段有一个错误： gitlab-ci.yaml

stages:
  - linter
  - build_pip
  - build
  - meta
  - code_quality
  - deploy

.except-tags:
  except:
    - tags

build_pip:build_dist:
  stage: build_pip
  # image: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/python:3.9-alpine
  image: python:3.9-alpine
  before_script:
    - apk update && apk add postgresql-dev gcc python3-dev musl-dev g++ jpeg-dev zlib-dev
    - pip install pip --upgrade
    - pip install -r requirements/production.txt --no-cache
  script:
    - python setup.py bdist_wheel
    - echo PIP_CI_JOB_ID=$CI_JOB_ID > PIP_CI_JOB_ID.env
  dependencies: []
  artifacts:
    expire_in: 1 hour
    paths:
      - dist/
      - version
    reports:
      dotenv: PIP_CI_JOB_ID.env
  extends:
    - .except-tags

meta:version:
  stage: meta
  needs:
    - job: build_pip:build_dist
      artifacts: true
  script:
    - cat version
  artifacts:
    expire_in: never
    paths:
      - version
  extends: .except-tags


build:build_api:
  stage: build
  image: registry.ml.bastion-tech.ru:8843/ansible/infrastructure/ansible_tools:2.9
  needs:
    - job: build_pip:build_dist
      artifacts: true
  before_script:
    - ansible-vault decrypt /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa --vault-password-file=${ANSIBLE_VAULT_PASSWORD}
  script:
    - |
      ansible-playbook -i /ansible/infrastructure/inventories/ml.inventory \
      --vault-password-file=${ANSIBLE_VAULT_PASSWORD} \
      --private-key /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa \
      -e ansible_ssh_user=deploy \
      -e smartconstructions_pip_ci_job_id=${PIP_CI_JOB_ID} \
      -e build=true -e smartconstructions_build_ref=${CI_COMMIT_BRANCH} \
      /ansible/infrastructure/ml_smartconstructions.yml
  tags:
    - linux-docker

deploy:deploy_api:
  stage: deploy
  image: registry.ml.bastion-tech.ru:8843/ansible/infrastructure/ansible_tools:2.9
  needs:
    - job: build_pip:build_dist
      artifacts: true
  when: manual
  only:
    - master
    - dev
  before_script:
    - ansible-vault decrypt /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa --vault-password-file=${ANSIBLE_VAULT_PASSWORD}
  script:
    - |
      ansible-playbook -i /ansible/infrastructure/inventories/ml.inventory \
      --vault-password-file=${ANSIBLE_VAULT_PASSWORD} \
      --private-key /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa \
      -e ansible_ssh_user=deploy \
      -e smartconstructions_pip_ci_job_id=${PIP_CI_JOB_ID} \
      -e run=true -e frontend_restart=true \
      /ansible/infrastructure/ml_smartconstructions.yml
  tags:
    - linux-docker

include:
  - local: .gitlab/ci/code-quality.yml

requirements/production.txt

djangorestframework==3.12.4
drf-extra-fields==3.1.1
djangorestframework-camel-case==1.2.0  # https://pypi.org/project/djangorestframework-camel-case/
Pillow==8.3.2
python-dateutil==2.8.2  # datetime formatting
psycopg2==2.9.1
opencv-python==4.5.3.56
drf-yasg==1.20.0
sentry-sdk==1.4.3
gunicorn==20.1.0
requests==2.26.0
yarl==1.7.0
googlemaps==4.5.3
django_redis==5.0.0
celery==5.2.0
channels==3.0.4
channels_redis==3.3.1

Answer 1

在您的日志中，我们可以看到以下错误：

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
  15 | #include <ffi.h>
    |          ^~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

此类错误表明您缺少头文件。

在 Alpine 中，ffi.h 文件应该是 libffi-dev 的一部分。试试这个：

apk add libffi-dev

Answer 2

也许您的用例会从使用 opencv-python-headless 中获益？它基本上是相同的库，由相同的人维护，但对 Xorg 和 GUI 包的依赖性较低。这意味着由于缺少软件包而导致的错误更少（即通过 apk 安装的东西更少）。

您可以在 development.txt 文件中使用 opencv-python 并将生产更改为无头版本。

来自文档：

These packages are smaller than the two other packages above because they do not contain any GUI functionality (not compiled with Qt / other GUI components). This means that the packages avoid a heavy dependency chain to X11 libraries and you will have for example smaller Docker images as a result. You should always use these packages if you do not use cv2.imshow et al. or you are using some other package (such as PyQt) than OpenCV to create your GUI.

Option 3 - Headless main modules package: pip install opencv-python-headless

Option 4 - Headless full package (contains both main modules and contrib/extra modules): pip install opencv-contrib-python-headless (check contrib/extra modules listing from OpenCV documentation)

除此之外，使用基于 Alpine 的图像 + Python + 科学工具真的是个好主意吗？考虑到一些 Python 工具对 GCC 等人的纠结程度，如果你在运行时遇到更多问题，即使这个图像设法构建，我也不会感到惊讶。

尝试使用 3.9-slim-bullseye，它是基于 debian 的映像。

Gitlab CI：opencv-python 构建轮失败

Gitlab CI: Failed building wheel for opencv-python

python

gitlab