Airflow 无法为具有自定义名称的插件文件夹导入插件

Question

我要处理一个共享的 Airflow 1.10 项目，所以我克隆了存储库，结构如下：

airflow
├── airflow.cfg
├── airflow.db
├── dags
│   ├── dags_here.py
│      
├── dags_conf
│   ├── some settings for dags
│   │   └── settings.py
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   └── todo.txt
|   |__ init.py
|
├── utilities
│   ├── __init__.py
│   └── some_name
│       ├── airflow
│       │   └── hooks.py
            └── sensors.py
            └── plugins.py
│       ├── common
│       │   ├some_other_code_files.py
│       ├── __init__.py
|
├── README.md
├── unittests.cfg
└── venv
    ├── lib64 -> lib
    └── pyvenv.cfg

但是当我尝试列出 dags 时，我遇到了这个错误：

plugins_manager.py:225} ERROR - No module named 'common' Traceback (most recent call last): File "/home/xxx/venv/lib/python3.8/site-packages/airflow/plugins_manager.py", line 218, in m = imp.load_source(namespace, filepath) File "/usr/lib/python3.8/imp.py", line 171, in load_source module = _load(spec) File "", line 702, in _load File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "/home/xxx/airflow/utilities/xxx/common/insert_obj.py", line 3, in from common.xxxxx import Class_name ModuleNotFoundError: No module named 'common' [2021-05-30 23:13:05,933] {plugins_manager.py:226} ERROR - Failed to import plugin /home/xxxx/airflow/utilities/xxxxx/common/dag_file.py

我在 may airflow.cfg 中尝试了以下配置，但总是出现相同的错误： plugins_folder=/home/xxx/airflow/utilities

在名为 plugins 的 Airflow 文件夹中添加一个带有 __init__.py 文件的父目录，并在其中复制实用程序： plugins_folder=/home/xxx/airflow/plugins

Answer 1

确认 Airflow 使用的环境变量或 airflow.cfg 设置的配置方式。

环境变量

AIRFLOW_HOME=<YOUR_PROJ>
AIRFLOW__CORE__DAGS_FOLDER=$AIRFLOW_HOME/dags
AIRFLOW__CORE__PLUGINS_FOLDER=$AIRFLOW_HOME/utilities

airflow.cfg

dags_folder = {AIRFLOW_HOME}/dags
plugins_folder = {AIRFLOW_HOME}/utilities

另外，看看PYTHONPATH中配置了什么：

import sys
from pprint import pprint

pprint(sys.path)

# [
#  '/ssddisk/project_name/dags',
#  '/ssddisk/project_name/config',
#  '/ssddisk/project_name/utilities',
#  ...
#  ]

有关详细信息，请参阅文档：https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html#creating-a-custom-operator

Airflow 无法为具有自定义名称的插件文件夹导入插件

Airflow Failed To Import Plugin for a plugin folder with custom name

airflow