在启动 celery 和 celery-beat 容器以防止 UniqueViolation 错误之前等待 django app 容器成为 运行?

Waiting for django app container to be running before starting celery and celery-beat container to prevent UniqueViolation errors?

编辑 1 2020 年 12 月 31 日

好吧,它不起作用,因为 django 服务仍然不健康 需要一些帮助


编辑 2020 年 12 月 31 日

错误 django_migrations 使用 django/postgresql/celery/celery-beat 和 DOCKER

的 UniqueViolation (celery-beat) 和 DuplicateTable (celery) 错误

我正在学习 docker-compose healthycheck,因为 depends_on 是不够的,因为容器只等待它依赖的容器启动

如果 django 网络应用的 celery 和 celery-beat“等待”运行ning 在 http://0.0.0.0:8000/ 上它应该可以防止下面的错误?

    ...
    django:
        restart: always
        container_name: django
    ...
        healthcheck:
            test: ["CMD", "curl", "-f", "http://0.0.0.0:8000"]
            interval: 30s
            timeout: 10s
            retries: 5
    celery:
        container_name: celery
        build: 
            context: ./app
            dockerfile: Dockerfile.dev
        command: celery -A core worker -l info
        volumes:
            - ./app:/usr/src/app
        env_file:
            - ./.env.dev
        depends_on:
            web:
                condition: service_healthy
            redis:
                condition: service_started

编辑

错误不一样auth_permission现在处理table


我开发了一个 Django 应用程序并尝试为 dbbackup 实现异步任务

堆栈:django/postgresql/celery/celery-beat和DOCKER(只有postgresql不在容器中)

我是 celery 和 celery-beat 的新手

当运行我的项目时,我有 2 个关于 celery 和 celery-beat 的错误我不明白

痕迹:

D:\Users\jl3\DOCKER\Apps\cafe_tropical>docker-compose -f docker-compose.dev.yml up
Creating network "cafe_tropical_default" with the default driver
Creating redis ... done                                                                                                                                                          Creating web         ... done                                                                                                                                                    Creating celery      ... done                                                                                                                                                    Creating celery-beat ... done                                                                                                                                                    Attaching to redis, celery, web, celery-beat
celery         | Waiting for postgres...
celery         | PostgreSQL started
celery-beat    | Waiting for postgres...
celery-beat    | PostgreSQL started
redis          | 1:C 30 Dec 2020 15:21:18.345 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis          | 1:C 30 Dec 2020 15:21:18.345 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis          | 1:C 30 Dec 2020 15:21:18.345 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis          | 1:M 30 Dec 2020 15:21:18.346 * Running mode=standalone, port=6379.
redis          | 1:M 30 Dec 2020 15:21:18.346 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis          | 1:M 30 Dec 2020 15:21:18.346 # Server initialized
redis          | 1:M 30 Dec 2020 15:21:18.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis          | 1:M 30 Dec 2020 15:21:18.346 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis          | 1:M 30 Dec 2020 15:21:18.348 * Ready to accept connections
web            | Waiting for postgres...
web            | PostgreSQL started
celery         | No changes detected
celery-beat    | No changes detected
web            | No changes detected
celery         | Operations to perform:
celery         |   Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery         | Running migrations:
celery         |   Applying contenttypes.0001_initial... OK
web            | Operations to perform:
web            |   Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
web            | Running migrations:
celery-beat    | Operations to perform:
celery-beat    |   Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery-beat    | Running migrations:
celery         |   Applying auth.0001_initial... OK
celery-beat    |   Applying auth.0001_initial...Traceback (most recent call last):
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat    |     return self.cursor.execute(sql)
celery-beat    | psycopg2.errors.UniqueViolation: ERREUR:  la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat    | DETAIL:  La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat    |
celery-beat    |
celery-beat    | The above exception was the direct cause of the following exception:
celery-beat    |
celery-beat    | Traceback (most recent call last):
celery-beat    |   File "manage.py", line 22, in <module>
web            |   Applying auth.0001_initial...Traceback (most recent call last):
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web            |     return self.cursor.execute(sql)
web            | psycopg2.errors.UniqueViolation: ERREUR:  la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web            | DETAIL:  La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web            |
web            |
web            | The above exception was the direct cause of the following exception:
web            |
web            | Traceback (most recent call last):
web            |   File "manage.py", line 22, in <module>
web            |     main()
web            |   File "manage.py", line 18, in main
celery-beat    |     main()
celery-beat    |   File "manage.py", line 18, in main
web            |     execute_from_command_line(sys.argv)
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
web            |     utility.execute()
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
web            |     self.fetch_command(subcommand).run_from_argv(self.argv)
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
web            |     self.execute(*args, **cmd_options)
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
web            |     output = self.handle(*args, **options)
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
web            |     res = handle_func(*args, **kwargs)
web            |   File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
web            |     post_migrate_state = executor.migrate(
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
web            |     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
web            |     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
web            |     state = migration.apply(state, schema_editor)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
web            |     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
web            |     schema_editor.create_model(model)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
web            |     self.execute(sql, params or None)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
web            |     cursor.execute(sql, params)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
web            |     return super().execute(sql, params)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
web            |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
web            |     return executor(sql, params, many, context)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
web            |     return self.cursor.execute(sql, params)
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
web            |     raise dj_exc_value.with_traceback(traceback) from exc_value
web            |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web            |     return self.cursor.execute(sql)
web            | django.db.utils.IntegrityError: ERREUR:  la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web            | DETAIL:  La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web            |
celery-beat    |     execute_from_command_line(sys.argv)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
celery-beat    |     utility.execute()
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
celery-beat    |     self.fetch_command(subcommand).run_from_argv(self.argv)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
celery-beat    |     self.execute(*args, **cmd_options)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
celery-beat    |     output = self.handle(*args, **options)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
celery-beat    |     res = handle_func(*args, **kwargs)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
celery-beat    |     post_migrate_state = executor.migrate(
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
celery-beat    |     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
celery-beat    |     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
celery-beat    |     state = migration.apply(state, schema_editor)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
celery-beat    |     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
celery-beat    |     schema_editor.create_model(model)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
celery-beat    |     self.execute(sql, params or None)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
celery-beat    |     cursor.execute(sql, params)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
celery-beat    |     return super().execute(sql, params)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
celery-beat    |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
celery-beat    |     return executor(sql, params, many, context)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-beat    |     return self.cursor.execute(sql, params)
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
celery-beat    |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-beat    |   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat    |     return self.cursor.execute(sql)
celery-beat    | django.db.utils.IntegrityError: ERREUR:  la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat    | DETAIL:  La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat    |
celery         |   Applying admin.0001_initial... OK
celery         |   Applying admin.0002_logentry_remove_auto_add... OK
celery         |   Applying admin.0003_logentry_add_action_flag_choices... OK
celery         |   Applying api.0001_initial... OK
celery         |   Applying contenttypes.0002_remove_content_type_name... OK
celery         |   Applying auth.0002_alter_permission_name_max_length... OK
celery         |   Applying auth.0003_alter_user_email_max_length... OK
celery         |   Applying auth.0004_alter_user_username_opts... OK
celery         |   Applying auth.0005_alter_user_last_login_null... OK
celery         |   Applying auth.0006_require_contenttypes_0002... OK
celery         |   Applying auth.0007_alter_validators_add_error_messages... OK
celery         |   Applying auth.0008_alter_user_username_max_length... OK
celery         |   Applying auth.0009_alter_user_last_name_max_length... OK
celery         |   Applying auth.0010_alter_group_name_max_length... OK
celery         |   Applying auth.0011_update_proxy_permissions... OK
celery         |   Applying authtoken.0001_initial... OK
celery         |   Applying authtoken.0002_auto_20160226_1747... OK
celery         |   Applying cafe.0001_initial... OK
celery         |   Applying django_celery_beat.0001_initial... OK
celery         |   Applying django_celery_beat.0002_auto_20161118_0346... OK
celery         |   Applying django_celery_beat.0003_auto_20161209_0049... OK
celery         |   Applying django_celery_beat.0004_auto_20170221_0000... OK
celery         |   Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
celery         |   Applying django_celery_beat.0006_auto_20180322_0932... OK
celery         |   Applying django_celery_beat.0007_auto_20180521_0826... OK
celery         |   Applying django_celery_beat.0008_auto_20180914_1922... OK
celery         |   Applying django_celery_beat.0006_auto_20180210_1226... OK
celery         |   Applying django_celery_beat.0006_periodictask_priority... OK
celery         |   Applying django_celery_beat.0009_periodictask_headers... OK
celery         |   Applying django_celery_beat.0010_auto_20190429_0326... OK
celery         |   Applying django_celery_beat.0011_auto_20190508_0153... OK
celery         |   Applying django_celery_beat.0012_periodictask_expire_seconds... OK
celery         |   Applying django_celery_beat.0013_auto_20200609_0727... OK
celery         |   Applying django_celery_beat.0014_remove_clockedschedule_enabled... OK
celery         |   Applying parameters.0001_initial... OK
celery         |   Applying sessions.0001_initial... OK
celery         |   Applying sites.0001_initial... OK
celery         |   Applying sites.0002_alter_domain_unique... OK
celery-beat    | celery beat v5.0.5 (singularity) is starting.
celery-beat    | __    -    ... __   -        _
celery-beat    | LocalTime -> 2020-12-30 15:21:35
celery-beat    | Configuration ->
celery-beat    |     . broker -> redis://redis:6379//
celery-beat    |     . loader -> celery.loaders.app.AppLoader
celery-beat    |     . scheduler -> celery.beat.PersistentScheduler
celery-beat    |     . db -> celerybeat-schedule
celery-beat    |     . logfile -> [stderr]@%INFO
celery-beat    |     . maxinterval -> 5.00 minutes (300s)
celery-beat    | [2020-12-30 15:21:35,216: INFO/MainProcess] beat: Starting...
celery-beat    | [2020-12-30 15:21:35,678: INFO/MainProcess] Scheduler: Sending due task hello (cafe.tasks.hello)
celery-beat    | [2020-12-30 15:21:35,724: INFO/MainProcess] Scheduler: Sending due task backup (cafe.tasks.backup)
web            | Watching for file changes with StatReloader
web            | Performing system checks...
web            |
web            | System check identified no issues (0 silenced).
web            | December 30, 2020 - 15:21:37
web            | Django version 2.2.5, using settings 'core.settings.dev'
web            | Starting development server at http://0.0.0.0:8000/
web            | Quit the server with CONTROL-C.
celery         | /usr/local/lib/python3.8/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
celery         | absolutely not recommended!
celery         |
celery         | Please specify a different user using the --uid option.
celery         |
celery         | User information: uid=0 euid=0 gid=0 egid=0
celery         |
celery         |   warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
celery         |
celery         |  -------------- celery@e53033835ccb v5.0.5 (singularity)
celery         | --- ***** -----
celery         | -- ******* ---- Linux-4.19.128-microsoft-standard-x86_64-with 2020-12-30 15:21:37
celery         | - *** --- * ---
celery         | - ** ---------- [config]
celery         | - ** ---------- .> app:         core:0x7fe23b6ba220
celery         | - ** ---------- .> transport:   redis://redis:6379//
celery         | - ** ---------- .> results:     redis://redis:6379/
celery         | - *** --- * --- .> concurrency: 4 (prefork)
celery         | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery         | --- ***** -----
celery         |  -------------- [queues]
celery         |                 .> celery           exchange=celery(direct) key=celery
celery         |
celery         |
celery         | [tasks]
celery         |   . cafe.tasks.backup
celery         |   . cafe.tasks.hello
celery         |   . core.celery.debug_task
celery         |
celery         | [2020-12-30 15:21:37,940: INFO/MainProcess] Connected to redis://redis:6379//
celery         | [2020-12-30 15:21:37,951: INFO/MainProcess] mingle: searching for neighbors
celery         | [2020-12-30 15:21:38,979: INFO/MainProcess] mingle: all alone
celery         | [2020-12-30 15:21:39,009: WARNING/MainProcess] /usr/local/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory

但对于 Django 网络应用程序容器

,所有迁移似乎都运行正确

我尝试在构建和 运行ning 容器之前删除 postgresql 数据库,所以我不明白 django_migrations table 怎么可能已经存在

可能是容器之间的依赖关系

我根据redis设置所有容器

version: '3.7'

services:
    web:
        restart: always
        container_name: web
        build: 
            context: ./app
            dockerfile: Dockerfile.dev
        restart: always
        command: python manage.py runserver 0.0.0.0:8000
        volumes:
            - ./app:/usr/src/app
        ports:
            - 8000:8000
        env_file:
            - ./.env.dev
        depends_on: 
            - redis
    redis:
        image: "redis:alpine"
    celery:
        build: 
            context: ./app
            dockerfile: Dockerfile.dev
        command: celery -A core worker -l info
        volumes:
            - ./app:/usr/src/app
        env_file:
            - ./.env.dev
        depends_on:
            - redis
    celery-beat:
        build: 
            context: ./app
            dockerfile: Dockerfile.dev
        command: celery -A core beat -l info
        volumes:
            - ./app:/usr/src/app
        env_file:
            - ./.env.dev
        depends_on:
            - redis

尝试使用 http://localhost:8000/(不要忘记末尾的“/”),它应该可以工作