Airflow db init 错误 - 无法为 GET /api/v1/connections 添加操作

Question

我正在尝试在 CentOS8 机器上安装带有 ansible 的 Airflow 2.0.1。 Python 版本 3.8.1。我按照 Airflow 文档中的建议制作了 pip 20.2.4。

我正在使用 postgresql，airflow db check 成功了。但是 db init 任务给出了以下错误。我手动尝试 airflow db init 但结果是一样的：

ERROR - Failed to add operation for GET /api/v1/connections
Traceback (most recent call last):
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apis/abstract.py", line 209, in add_paths
    self.add_operation(path, method)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apis/abstract.py", line 162, in add_operation
    operation = make_operation(
  File "/opt/airflow/lib/python3.8/site-packages/connexion/operations/__init__.py", line 8, in make_operation
    return spec.operation_cls.from_spec(spec, *args, **kwargs)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/operations/openapi.py", line 128, in from_spec
    return cls(
  File "/opt/airflow/lib/python3.8/site-packages/connexion/operations/openapi.py", line 75, in __init__
    super(OpenAPIOperation, self).__init__(
  File "/opt/airflow/lib/python3.8/site-packages/connexion/operations/abstract.py", line 96, in __init__
    self._resolution = resolver.resolve(self)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/resolver.py", line 40, in resolve
    return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/resolver.py", line 66, in resolve_function_from_operation_id
    raise ResolverError(str(e), sys.exc_info())
connexion.exceptions.ResolverError: <ResolverError: columns>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/airflow/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/airflow/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
    db.initdb()
  File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/db.py", line 549, in initdb
    upgradedb()
  File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/db.py", line 684, in upgradedb
    command.upgrade(config, 'heads')
  File "/opt/airflow/lib/python3.8/site-packages/alembic/command.py", line 294, in upgrade
    script.run_env()
  File "/opt/airflow/lib/python3.8/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/airflow/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/airflow/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/lib/python3.8/site-packages/airflow/migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/opt/airflow/lib/python3.8/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/airflow/lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/airflow/lib/python3.8/site-packages/alembic/runtime/migration.py", line 561, in run_migrations
    step.migration_fn(**kw)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 314, in upgrade
    remap_permissions()
  File "/opt/airflow/lib/python3.8/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 289, in remap_permissions
    appbuilder = create_app(config={'FAB_UPDATE_PERMS': False}).appbuilder
  File "/opt/airflow/lib/python3.8/site-packages/airflow/www/app.py", line 120, in create_app
    init_api_connexion(flask_app)
  File "/opt/airflow/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 171, in init_api_connexion
    api_bp = connexion_app.add_api(
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apps/flask_app.py", line 57, in add_api
    api = super(FlaskApp, self).add_api(specification, **kwargs)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apps/abstract.py", line 144, in add_api
    api = self.api_cls(specification,
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apis/abstract.py", line 111, in __init__
    self.add_paths()
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apis/abstract.py", line 216, in add_paths
    self._handle_add_operation_error(path, method, err.exc_info)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/apis/abstract.py", line 231, in _handle_add_operation_error
    raise value.with_traceback(traceback)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/resolver.py", line 61, in resolve_function_from_operation_id
    return self.function_resolver(operation_id)
  File "/opt/airflow/lib/python3.8/site-packages/connexion/utils.py", line 111, in get_function_from_name
    module = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/lib/python3.8/site-packages/airflow/api_connexion/endpoints/connection_endpoint.py", line 26, in <module>
    from airflow.api_connexion.schemas.connection_schema import (
  File "/opt/airflow/lib/python3.8/site-packages/airflow/api_connexion/schemas/connection_schema.py", line 42, in <module>
    class ConnectionSchema(ConnectionCollectionItemSchema):  # pylint: disable=too-many-ancestors
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow/schema.py", line 121, in __new__
    klass._declared_fields = mcs.get_declared_fields(
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 94, in get_declared_fields
    fields.update(mcs.get_auto_fields(fields, converter, opts, dict_cls))
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 104, in get_auto_fields
    {
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 105, in <dictcomp>
    field_name: field.create_field(
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 28, in create_field
    return converter.field_for(model, column_name, **self.field_kwargs)
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/convert.py", line 171, in field_for
    return self.property2field(prop, **kwargs)
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/convert.py", line 146, in property2field
    field_class = field_class or self._get_field_class_for_property(prop)
  File "/opt/airflow/lib/python3.8/site-packages/marshmallow_sqlalchemy/convert.py", line 210, in _get_field_class_for_property
    column = prop.columns[0]
  File "/opt/airflow/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 1220, in __getattr__
    return self._fallback_getattr(key)
  File "/opt/airflow/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 1194, in _fallback_getattr
    raise AttributeError(key)
AttributeError: columns

Answer 1

我在运行airflow db init之前设置了AIRFLOW_HOME和AIRFLOW__CORE__SQL_ALCHEMY_CONN环境。

剧本版本

environment:
    AIRFLOW_HOME: "{{ airflow_app_home }}"
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: "{{ airflow_database_conn }}"

Answer 2

我遇到了同样的问题，临时修复是使用 sqlalchemy < 1.4。也许这对你有用。

Answer 3

对我有用的是设置 AIRFLOW_HOME 和 AIRFLOW__CORE__SQL_ALCHEMY_CONN 以及 pip install SQLAlchemy==1.3.24

Answer 4

对我和 Airflow 2.3.1 升级 marshmallow-sqlalchemy 有帮助。

pip install -U marshmallow-sqlalchemy

Airflow db init 错误 - 无法为 GET /api/v1/connections 添加操作

Airflow db init ERROR - Failed to add operation for GET /api/v1/connections

centos

airflow

centos8