运行 Elastic Beanstalk 上带有主管的后台 huey 任务队列

Question

我正在尝试运行配置文件中的 huey task queue on elastic beanstalk that is needed by my Flask app. But there is no built in way to run huey as a daemon process. The author of huey has advised to run huey with supervisor (this link) and since elastic beanstalk already uses supervisor, I thought we could just add the program to be managed by supervisor. But I am not sure how to do this programatically. Currently, I am using the container_commands (ref link) 键到运行这个但是弹性 beantalk 在一段时间后给我一个超时错误运行 s 在前台。下面是我正在使用的配置文件。

packages:
  yum:
    gcc: []
    gcc-c++: []
    gcc-gfortran: []
    htop: []
    make: []
    wget: []
    atlas-devel: []
    lapack-devel: []
commands:
  01enable_swap:
    command:
      - sudo dd if=/dev/zero of=/var/swap1 bs=1M count=1024
      - sudo mkswap /var/swap1
      - sudo chmod 644 /var/swap1
      - sudo swapon /var/swap1
    cwd: /home/ec2-user
  02install_redis:
    command:
      - wget "http://download.redis.io/redis-stable.tar.gz"
      - tar -xvzf redis-stable.tar.gz
      - rm redis-stable.tar.gz
      - cd redis-stable
      - sudo make
      - sudo make install
    cwd: /home/ec2-user
container_commands:
  01download_nltk_packages:
    command: "python install_resources.py"
  02run_redis:
    command: "redis-server --host 127.0.0.1 --port 6379 --daemonize yes"
  03run_huey:
    command: "huey_consumer jupiter.huey"

这是我想要实现的目标：
1. 部署我的 Flask 应用程序时，huey 应该运行作为后台进程。
2. supervisor 应该处理 huey 进程的自动 start/stop。

Answer 1

我通过在名为 002_supervisor.conf 的 ebextensions 文件中执行以下操作解决了这个问题。这是针对 django 的，但我相信它可以适用于 flask。

创建主管配置文件
创建主管init.d文件
创建一个 huey.conf 文件以供主管加载

files:
   /usr/local/etc/supervisord.conf:
    mode: "000755"
    owner: root
    group: root
    content: |
        [unix_http_server]
        file=/tmp/supervisor.sock   ; (the path to the socket file)

        [supervisord]
        logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
        pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
        nodaemon=false               ; (start in foreground if true;default false)

        [rpcinterface:supervisor]
        supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

        [supervisorctl]
        serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

        [include]
        files = /usr/local/etc/*.conf

        [inet_http_server]
        port = 127.0.0.1:9001
   /etc/init.d/supervisord:
    mode: "000755"
    owner: root
    group: root
    content: |
        #!/bin/bash

        # Source function library
        . /etc/rc.d/init.d/functions

        # Source system settings
        if [ -f /etc/sysconfig/supervisord ]; then
            . /etc/sysconfig/supervisord
        fi

        # Path to the supervisorctl script, server binary,
        # and short-form for messages.
        supervisorctl=/usr/local/bin/supervisorctl
        supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
        prog=supervisord
        pidfile=${PIDFILE-/tmp/supervisord.pid}
        lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
        STOP_TIMEOUT=${STOP_TIMEOUT-60}
        OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
        RETVAL=0

        start() {
            echo -n $"Starting $prog: "
            daemon --pidfile=${pidfile} $supervisord $OPTIONS
            RETVAL=$?
            echo
            if [ $RETVAL -eq 0 ]; then
                touch ${lockfile}
                $supervisorctl $OPTIONS status
            fi
            return $RETVAL
        }

        stop() {
            echo -n $"Stopping $prog: "
            killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
            RETVAL=$?
            echo
            [ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
        }

        reload() {
            echo -n $"Reloading $prog: "
            LSB=1 killproc -p $pidfile $supervisord -HUP
            RETVAL=$?
            echo
            if [ $RETVAL -eq 7 ]; then
                failure $"$prog reload"
            else
                $supervisorctl $OPTIONS status
            fi
        }

        restart() {
            stop
            start
        }

        case "" in
            start)
                start
                ;;
            stop)
                stop
                ;;
            status)
                status -p ${pidfile} $supervisord
                RETVAL=$?
                [ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
                ;;
            restart)
                restart
                ;;
            condrestart|try-restart)
                if status -p ${pidfile} $supervisord >&/dev/null; then
                  stop
                  start
                fi
                ;;
            force-reload|reload)
                reload
                ;;
            *)
                echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
                RETVAL=2
            esac

            exit $RETVAL
   "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_huey.sh" :
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Get django environment variables
      env=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
      env=${env%?}

      # Create huey configuration script
      hueyconf="[program:huey]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/current/app/production.py run_huey
      user=nobody
      numprocs=1
      stdout_logfile=/var/log/huey.log
      stderr_logfile=/var/log/huey.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true
      environment=$env"

      # Create the celery supervisord conf script
      echo "$hueyconf" | tee /usr/local/etc/huey.conf

      # Update supervisord in cache without restarting all services
      /usr/local/bin/supervisorctl reread
      /usr/local/bin/supervisorctl update

      # Start/Restart huey through supervisord
      /usr/local/bin/supervisorctl -c /usr/local/etc/supervisord.conf restart huey

commands:
  01_start_supervisor:
    command: '/etc/init.d/supervisord restart'
    leader_only: true

运行 Elastic Beanstalk 上带有主管的后台 huey 任务队列

Run huey task queue in the background with supervisor on Elastic Beanstalk

python

amazon-ec2

amazon-web-services

amazon-elastic-beanstalk

python-huey