Airflow DAG 文件夹不可见
Airflow DAG folder is not visible
我刚刚在 ubuntu 中安装了 Airflow,我正在按照教程创建 DAG。成功安装气流后,我在气流主页中创建了 dags
文件夹,并创建了一个示例 DAG 文件作为 sample.py
。但我实际上没有看到 dags 文件夹
yash@DESKTOP-TUTMG4J:~/airflow$ ls -l
total 316
-rw-rw-rw- 1 yash yash 0 Jun 12 19:26 airflow-webserver.err
-rw-rw-rw- 1 yash yash 140 Jun 12 19:29 airflow-webserver.log
-rw-rw-rw- 1 yash yash 222 Jun 12 19:29 airflow-webserver.out
-rw-r--r-- 1 yash yash 5 Jun 13 14:53 airflow-webserver.pid
-rw-rw-rw- 1 yash yash 36462 Jun 12 17:02 airflow.cfg
-rw-r--r-- 1 yash yash 270336 Jun 13 14:54 airflow.db
drwxrwxrwx 1 yash yash 512 Jun 12 21:57 logs
-rw-rw-rw- 1 yash yash 2578 Jun 12 17:02 unittests.cfg
yash@DESKTOP-TUTMG4J:~/airflow$
但是当我启动 airflow webserver
它说 Filling up the DagBag from /home/yash/airflow/dags
yash@DESKTOP-TUTMG4J:~/airflow$ airflow initdb
DB: sqlite:////home/yash/airflow/airflow.db
[2020-06-13 14:51:35,133] {db.py:378} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Done.
yash@DESKTOP-TUTMG4J:~/airflow$ airflow webserver
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-06-13 14:53:22,260] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:22,260] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-06-13 14:53:23 +0530] [5308] [INFO] Starting gunicorn 19.4.0
[2020-06-13 14:53:23 +0530] [5308] [INFO] Listening at: http://0.0.0.0:8080 (5308)
[2020-06-13 14:53:23 +0530] [5308] [INFO] Using worker: sync
[2020-06-13 14:53:23 +0530] [5313] [INFO] Booting worker with pid: 5313
[2020-06-13 14:53:23 +0530] [5314] [INFO] Booting worker with pid: 5314
[2020-06-13 14:53:23 +0530] [5315] [INFO] Booting worker with pid: 5315
[2020-06-13 14:53:23 +0530] [5327] [INFO] Booting worker with pid: 5327
[2020-06-13 14:53:23,678] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,679] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,836] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,837] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,949] {__init__.py:51} INFO - Using executor SequentialExecutor
当然我也没有在气流中看到我的 DAG UI。谁能告诉我我做错了什么以及气流如何从文件夹中挑选我的 DAGS。
看来您实际上并没有在 Airflow 的主页中创建 dags
目录。在你的例子中,Airflow 的家是 /home/yash/airflow/
,显然那里没有 dags
目录。
Airflow 的日志消息 Filling up the DagBag from /home/yash/airflow/dags
可能有些误导。其实就是Airflows在那个路径下找DAG脚本,但不代表这个路径存在
创建目录:
mkdir /home/yash/airflow/dags
然后再次尝试创建 DAG 脚本。
我刚刚在 ubuntu 中安装了 Airflow,我正在按照教程创建 DAG。成功安装气流后,我在气流主页中创建了 dags
文件夹,并创建了一个示例 DAG 文件作为 sample.py
。但我实际上没有看到 dags 文件夹
yash@DESKTOP-TUTMG4J:~/airflow$ ls -l
total 316
-rw-rw-rw- 1 yash yash 0 Jun 12 19:26 airflow-webserver.err
-rw-rw-rw- 1 yash yash 140 Jun 12 19:29 airflow-webserver.log
-rw-rw-rw- 1 yash yash 222 Jun 12 19:29 airflow-webserver.out
-rw-r--r-- 1 yash yash 5 Jun 13 14:53 airflow-webserver.pid
-rw-rw-rw- 1 yash yash 36462 Jun 12 17:02 airflow.cfg
-rw-r--r-- 1 yash yash 270336 Jun 13 14:54 airflow.db
drwxrwxrwx 1 yash yash 512 Jun 12 21:57 logs
-rw-rw-rw- 1 yash yash 2578 Jun 12 17:02 unittests.cfg
yash@DESKTOP-TUTMG4J:~/airflow$
但是当我启动 airflow webserver
它说 Filling up the DagBag from /home/yash/airflow/dags
yash@DESKTOP-TUTMG4J:~/airflow$ airflow initdb
DB: sqlite:////home/yash/airflow/airflow.db
[2020-06-13 14:51:35,133] {db.py:378} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Done.
yash@DESKTOP-TUTMG4J:~/airflow$ airflow webserver
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-06-13 14:53:22,260] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:22,260] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-06-13 14:53:23 +0530] [5308] [INFO] Starting gunicorn 19.4.0
[2020-06-13 14:53:23 +0530] [5308] [INFO] Listening at: http://0.0.0.0:8080 (5308)
[2020-06-13 14:53:23 +0530] [5308] [INFO] Using worker: sync
[2020-06-13 14:53:23 +0530] [5313] [INFO] Booting worker with pid: 5313
[2020-06-13 14:53:23 +0530] [5314] [INFO] Booting worker with pid: 5314
[2020-06-13 14:53:23 +0530] [5315] [INFO] Booting worker with pid: 5315
[2020-06-13 14:53:23 +0530] [5327] [INFO] Booting worker with pid: 5327
[2020-06-13 14:53:23,678] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,679] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,836] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,837] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,949] {__init__.py:51} INFO - Using executor SequentialExecutor
当然我也没有在气流中看到我的 DAG UI。谁能告诉我我做错了什么以及气流如何从文件夹中挑选我的 DAGS。
看来您实际上并没有在 Airflow 的主页中创建 dags
目录。在你的例子中,Airflow 的家是 /home/yash/airflow/
,显然那里没有 dags
目录。
Airflow 的日志消息 Filling up the DagBag from /home/yash/airflow/dags
可能有些误导。其实就是Airflows在那个路径下找DAG脚本,但不代表这个路径存在
创建目录:
mkdir /home/yash/airflow/dags
然后再次尝试创建 DAG 脚本。