为什么 Dramatiq 无法在 Docker-容器中启动？

Question

我正在尝试使用 Dramatiq 作为工作库来做一个分布式任务队列 Django 项目。从开发环境中执行时，我能够运行项目没有错误，但是在尝试运行内置的 docker-容器时得到神秘的 FileNotFoundError。我运行想出了为什么会这样。

错误的回溯是：

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/dist-packages/django_dramatiq/management/commands/rundramatiq.py", line 83, in handle
    os.execvp(executable_path, process_args)
  File "/usr/lib/python3.5/os.py", line 615, in execvp
    _execvpe(file, args)
  File "/usr/lib/python3.5/os.py", line 660, in _execvpe
    raise last_exc.with_traceback(tb)
  File "/usr/lib/python3.5/os.py", line 650, in _execvpe
    exec_func(fullname, *argrest)
FileNotFoundError: [Errno 2] No such file or directory

我制作了一个 Django 项目框架，其中包含一个 Dramatiq 任务来重现该问题。可以从这里克隆存储库：https://bitbucket.org/AvareaToniN/dramatiqexample/src/master/，这里是该存储库的相关部分：

dramatiqexample
│   .gitignore
│   circus.ini
│   Dockerfile
│   entrypoint.sh
│   manage.py
│   requirements.txt
│
├───dramatiqexample
│       settings.py
│       urls.py
│       wsgi.py
│       __init__.py
│
└───exampleapp
    │   admin.py
    │   apps.py
    │   models.py
    │   tasks.py
    │   tests.py
    │   views.py
    │   __init__.py
    │
    └───migrations
            __init__.py

要求

# requirements.txt
Django==2.0.7
dramatiq[redis, watch]
django_dramatiq==0.4.0
apscheduler==3.5.1

马戏团配置

# circus.ini
[circus]
check_delay = 5

[watcher:gunicorn]
cmd = /usr/local/bin/gunicorn
args = -b 0.0.0.0:8000 -w 2 dramatiqexample.wsgi
numprocesses = 1
autostart = true
max_retry = -1
priority = 500

[watcher:dramatiq]
cmd = python
args = manage.py rundramatiq --processes 6 --no-reload --path ./dramatiqexample
autostart = true
max_retry = -1
priority = 50

[watcher:redis]
cmd = redis-server
autostart = true
max_retry = -1
priority = 200

[env:redis]
PATH = $PATH: /usr/local/bin/

Docker 文件

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y software-properties-common curl \
    && add-apt-repository ppa:ubuntugis/ubuntugis-unstable \
    && apt-get update \
    && apt-get install -y \
    python3-pip \
    libssl-dev \
    libffi-dev \
    libzmq-dev \
    libevent-dev \
    python3-gdal \
    gcc \
    g++ \
    build-essential \
    openssl \
    nginx \
    apt-transport-https \
    wget \
    && update-alternatives --install /usr/bin/python python /usr/bin/python3 10 \
    && update-alternatives --install /usr/bin/pip    pip    /usr/bin/pip3    10 \
    && rm -rf /var/lib/apt/lists/*

# redis
RUN wget http://download.redis.io/redis-stable.tar.gz
RUN tar xvzf redis-stable.tar.gz
WORKDIR redis-stable
RUN make
RUN make install
WORKDIR /
# End of redis

RUN echo openssl version

RUN pip install pip --upgrade
#https://github.com/circus-tent/circus/issues/1056
RUN pip install 'tornado==4.5.3'
RUN pip install gunicorn circus

ADD requirements.txt /

RUN pip install -r requirements.txt
ADD . /
RUN python manage.py makemigrations
RUN python manage.py migrate
RUN python manage.py collectstatic --noinput

EXPOSE 8000

RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh

#!/bin/bash

exec circusd circus.ini --log-level debug

exec "$@";

Django 设置：

# dramatiqexample/settings.py
import os
import redis
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'm7m#nre09wdf(jbe%#h+3o9l8j_%v7h87cc!rd@ow6tfy=-1'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []


# Application definition

INSTALLED_APPS = [
    'django_dramatiq',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'exampleapp.apps.ExampleappConfig',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'dramatiqexample.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')]
        ,
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'dramatiqexample.wsgi.application'


# Database
# https://docs.djangoproject.com/en/2.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Password validation
# https://docs.djangoproject.com/en/2.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/2.0/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.0/howto/static-files/

PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')
STATIC_URL = '/static/'

# DRAMATIQ CONFIG
DRAMATIQ_REDIS_URL = os.getenv('REDIS_URL', 'redis://127.0.0.1:6379/0')
DRAMATIQ_BROKER = {
    "BROKER": "dramatiq.brokers.redis.RedisBroker",
    "OPTIONS": {
        "connection_pool": redis.ConnectionPool.from_url(DRAMATIQ_REDIS_URL),
    },
    "MIDDLEWARE": [
        "dramatiq.middleware.Prometheus",
        "dramatiq.middleware.AgeLimit",
        "dramatiq.middleware.TimeLimit",
        "dramatiq.middleware.Callbacks",
        "dramatiq.middleware.Retries",
        "django_dramatiq.middleware.AdminMiddleware",
        "django_dramatiq.middleware.DbConnectionsMiddleware",
    ]
}

# Defines which database should be used to persist Task objects when the
# AdminMiddleware is enabled.  The default value is "default".
DRAMATIQ_TASKS_DATABASE = "default"

网址

#dramatiqexample/urls.py
from django.contrib import admin
from django.urls import path

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from exampleapp.tasks import minutely_tasks

urlpatterns = [
    path('admin/', admin.site.urls),

]

# 
SCHEDULER = BackgroundScheduler()
SCHEDULER.add_job(
    minutely_tasks.send,
    CronTrigger.from_crontab("* * * * *"),
)
try:
    SCHEDULER.start()
except KeyboardInterrupt:
    SCHEDULER.shutdown()

测试任务

# exampleapp/tasks.py
import dramatiq
import logging
from datetime import datetime

logger = logging.getLogger(__name__)

@dramatiq.actor
def minutely_tasks():
    logger.warning(datetime.now())

从 IDE 开始，命令 "python manage.py runserver" 和 "python manage.py rundramatiq" 工作得很好，但由于某些原因，它们在 dockerized circus-process 中失败了。

Answer 1

我遇到了同样的问题。这是与 dramatiq 的安装相关的问题。事实上，在我的例子中，脚本将在 /usr/local/lib 中执行 /usr/lib 中的 "dramatiq" 文件。所以我找到了一个解决方案：创建一个虚拟环境。

为什么 Dramatiq 无法在 Docker-容器中启动？

Why does Dramatiq fail to start in Docker-container?

python

django

docker

circusd