运行 conda 环境中的气流和 systemd 无法启动

Question

我想通过 conda 安装气流并使用 systemd 控制 Ubuntu 上的气流。我已经能够通过以下步骤将 airflow 成功安装到 conda 环境中，但是我无法正确配置 systemd 以使用 airflow。

详情：

Ubuntu 18.04
Python 3.7.4
气流 1.10.4

我看过SO #52310217，但问题仍未解决。

以下是我从命令行安装和运行 airflow 所采取的步骤：

$ conda create --name airflow -c conda-forge airflow
$ conda activate airflow
$ export AIRFLOW_HOME=~/airflow
$ airflow initdb

$ airflow scheduler
$ airflow webserver

从这里开始，我在 /etc/systemd/system/ 中创建了以下文件：

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

创建上述文件后，我发出了以下命令，但没有任何反应。

sudo systemctl daemon-reload
sudo systemctl enable airflow-scheduler
sudo systemctl start airflow-scheduler
sudo systemctl enable airflow-webserver
sudo systemctl start airflow-webserver

如果我直接在终端 window 中输入 ExecStart 命令，airflow scheduler 和 airflow webserver 会按预期启动。因此，我认为这不是气流问题，而是 systemd 配置中缺少的东西。

我不确定从中得到什么，但这里是 airflow-scheduler 和 airflow-webserver 的状态。

[curtis:~/airflow/logs/scheduler] [airflow] 4 $ sudo systemctl status airflow-scheduler
● airflow-scheduler.service - Airflow scheduler daemon
   Loaded: loaded (/etc/systemd/system/airflow-scheduler.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:35 PDT; 3s ago
  Process: 5623 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler (code=exited, status=217/USER)
 Main PID: 5623 (code=exited, status=217/USER)

[curtis:~/airflow/logs/scheduler] [airflow] 3 $ sudo systemctl status airflow-webserver
● airflow-webserver.service - Airflow webserver daemon
   Loaded: loaded (/etc/systemd/system/airflow-webserver.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:46 PDT; 895ms ago
  Process: 5805 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver
 Main PID: 5805 (code=exited, status=217/USER)

Answer 1

我发现了这个问题，并想post在这里为遇到同样问题的其他人提供这个。

User 应该是您当前的用户名，或者安装 Airflow 的用户名。在我的例子中是 "curtis"
Group应该是你当前所在的组，或者安装Airflow的用户所在的组。在我的例子中是 "curtis"
Environment 是您的 conda 环境的路径。就我而言，那是 PATH=/home/curtis/miniconda3/envs/airflow/bin

我已经包含了我更正的 systemd 文件供您参考。

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

运行 conda 环境中的气流和 systemd 无法启动

Running airflow within a conda environment and with systemd fails to launch

python

conda

airflow

airflow-scheduler

ubuntu-18.04