气流：SQLALCHEMY_DATABASE_URI 和 SQLALCHEMY_BINDS 均未设置
Airflow: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set
我已经在本地 mac 上安装了 airflow 2.0，支持 sql_alchemy_conn 的 PostgresSQL。我的 DAG 运行 很好 + 在 sqla 数据库中维护元数据，但是任何 airflow 用户命令错误 table/views 未创建角色。
我已经尝试设置和重置 airflow resetdb，但这些表在 sql_alchemy_conn 数据库中仍然不可用。
psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
[2019-01-11 11:45:58,223] {__init__.py:51} INFO - Using executor SequentialExecutor
/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:774: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
Please make sure to build the frontend in static/ directory and restart the server
[2019-01-11 11:45:58,450] {manager.py:525} WARNING - No user yet created, use fabmanager command to do it.
[2019-01-11 11:45:59,117] {__init__.py:286} INFO - Filling up the DagBag from /Users/deepaksaroha/airflow/dags
[2019-01-11 11:46:00,114] {security.py:435} INFO - Start syncing user roles.
[2019-01-11 11:46:00,149] {security.py:185} INFO - Initializing permissions for role:Viewer in the database.
[2019-01-11 11:46:00,274] {security.py:185} INFO - Initializing permissions for role:User in the database.
[2019-01-11 11:46:00,397] {security.py:185} INFO - Initializing permissions for role:Op in the database.
[2019-01-11 11:46:00,487] {security.py:344} INFO - Fetching a set of all permission, view_menu from FAB meta-table
Traceback (most recent call last):
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/bin/airflow", line 32, in <module>
args.func(args)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/bin/cli.py", line 1353, in users
appbuilder = cached_appbuilder()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 214, in cached_appbuilder
cached_app(config, testing)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 207, in cached_app
app, _ = create_app(config, session, testing)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 167, in create_app
security_manager.sync_roles()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 443, in sync_roles
self.create_custom_dag_permission_view()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 391, in create_custom_dag_permission_view
all_perm_views = set([role.permission_view_id for role in all_perm_view_by_user])
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
return self._execute_and_instances(context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
^
[SQL: 'SELECT ab_permission_view_role.id AS ab_permission_view_role_id, ab_permission_view_role.permission_view_id AS ab_permission_view_role_permission_view_id, ab_permission_view_role.role_id AS ab_permission_view_role_role_id \nFROM ab_permission_view_role JOIN ab_permission_view ON ab_permission_view.id = ab_permission_view_role.permission_view_id JOIN ab_view_menu ON ab_view_menu.id = ab_permission_view.view_menu_id \nWHERE ab_permission_view_role.role_id = %(role_id_1)s AND ab_permission_view.view_menu_id != %(view_menu_id_1)s'] [parameters: {'role_id_1': 4, 'view_menu_id_1': 51}] (Background on this error at: http://sqlalche.me/e/f405)
我希望这些表应该在后端某处创建。我想使用 airflow users 命令通过脚本自动创建用户，但存在错误，上帝保佑我:)
我得到了这个解决方案。发生这种情况是因为 ab_* tables 不是在 airflow initdb 时创建的。所有这些表都用于基于角色的访问控制——RBAC。
要获得这些表格，请按照说明操作：
- 编辑
airflow.cfg
[webserver]
rbac = True
- 运行
airflow initdb 创建这些缺失的表。
这解决了我的问题。
您可以在 airflow 源代码中修补 app.py：
在数据库初始化 db = SQLA(app).
之前从 SQL_ALCHEMY_CONN 在 app.config 中设置 SQLALCHEMY_DATABASE_URI
这是 app.py 文件的路径：
/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py
这是你应该包括的行：
app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')
@appleboy 的选择答案对我不起作用，但@Nando_Quintana 的解决方案对我有用。
执行以下操作以将他的修复作为补丁应用
cd path/to/python/site-packages/airflow/www_rbac/
patch <<EOF
--- _app.py2019-09-25 17:18:59.127378849 +0000
+++ app.py 2019-09-25 17:18:53.467413221 +0000
@@ -53,6 +53,7 @@
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['APP_NAME'] = app_name
app.config['TESTING'] = testing
+ app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')
csrf.init_app(app)
EOF
在 Python 3.6.8 和 Airflow 1.10.2 上测试
airflow 1.10.12 中最常见的原因是添加了一个 $AIRFLOW_HOME/webserver_config.py 文件来覆盖默认值。
如果要添加 webserver_config.py，它必须包含
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
喜欢 default webserver_config.py.
原则上，如果您在 airflow.cfg 中设置 rbac = True，删除 webserver_config.py 并执行任何命令，例如 airflow list_users 然后 airflow 将为创建一个 webserver_config.py 文件您使用适当的 SQLALCHEMY_DATABASE_URI = ... 可以在之后修改。
我已经在本地 mac 上安装了 airflow 2.0，支持 sql_alchemy_conn 的 PostgresSQL。我的 DAG 运行 很好 + 在 sqla 数据库中维护元数据，但是任何 airflow 用户命令错误 table/views 未创建角色。
我已经尝试设置和重置 airflow resetdb，但这些表在 sql_alchemy_conn 数据库中仍然不可用。
psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
[2019-01-11 11:45:58,223] {__init__.py:51} INFO - Using executor SequentialExecutor
/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:774: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
Please make sure to build the frontend in static/ directory and restart the server
[2019-01-11 11:45:58,450] {manager.py:525} WARNING - No user yet created, use fabmanager command to do it.
[2019-01-11 11:45:59,117] {__init__.py:286} INFO - Filling up the DagBag from /Users/deepaksaroha/airflow/dags
[2019-01-11 11:46:00,114] {security.py:435} INFO - Start syncing user roles.
[2019-01-11 11:46:00,149] {security.py:185} INFO - Initializing permissions for role:Viewer in the database.
[2019-01-11 11:46:00,274] {security.py:185} INFO - Initializing permissions for role:User in the database.
[2019-01-11 11:46:00,397] {security.py:185} INFO - Initializing permissions for role:Op in the database.
[2019-01-11 11:46:00,487] {security.py:344} INFO - Fetching a set of all permission, view_menu from FAB meta-table
Traceback (most recent call last):
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/bin/airflow", line 32, in <module>
args.func(args)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/bin/cli.py", line 1353, in users
appbuilder = cached_appbuilder()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 214, in cached_appbuilder
cached_app(config, testing)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 207, in cached_app
app, _ = create_app(config, session, testing)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 167, in create_app
security_manager.sync_roles()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 443, in sync_roles
self.create_custom_dag_permission_view()
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 391, in create_custom_dag_permission_view
all_perm_views = set([role.permission_view_id for role in all_perm_view_by_user])
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
return self._execute_and_instances(context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
^
[SQL: 'SELECT ab_permission_view_role.id AS ab_permission_view_role_id, ab_permission_view_role.permission_view_id AS ab_permission_view_role_permission_view_id, ab_permission_view_role.role_id AS ab_permission_view_role_role_id \nFROM ab_permission_view_role JOIN ab_permission_view ON ab_permission_view.id = ab_permission_view_role.permission_view_id JOIN ab_view_menu ON ab_view_menu.id = ab_permission_view.view_menu_id \nWHERE ab_permission_view_role.role_id = %(role_id_1)s AND ab_permission_view.view_menu_id != %(view_menu_id_1)s'] [parameters: {'role_id_1': 4, 'view_menu_id_1': 51}] (Background on this error at: http://sqlalche.me/e/f405)
我希望这些表应该在后端某处创建。我想使用 airflow users 命令通过脚本自动创建用户，但存在错误，上帝保佑我:)
我得到了这个解决方案。发生这种情况是因为 ab_* tables 不是在 airflow initdb 时创建的。所有这些表都用于基于角色的访问控制——RBAC。
要获得这些表格，请按照说明操作：
- 编辑
airflow.cfg
[webserver]
rbac = True
- 运行
airflow initdb创建这些缺失的表。
这解决了我的问题。
您可以在 airflow 源代码中修补 app.py：
在数据库初始化 db = SQLA(app).
SQL_ALCHEMY_CONN 在 app.config 中设置 SQLALCHEMY_DATABASE_URI
这是 app.py 文件的路径：
/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py
这是你应该包括的行：
app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')
@appleboy 的选择答案对我不起作用，但@Nando_Quintana 的解决方案对我有用。
执行以下操作以将他的修复作为补丁应用
cd path/to/python/site-packages/airflow/www_rbac/
patch <<EOF
--- _app.py2019-09-25 17:18:59.127378849 +0000
+++ app.py 2019-09-25 17:18:53.467413221 +0000
@@ -53,6 +53,7 @@
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['APP_NAME'] = app_name
app.config['TESTING'] = testing
+ app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')
csrf.init_app(app)
EOF
在 Python 3.6.8 和 Airflow 1.10.2 上测试
airflow 1.10.12 中最常见的原因是添加了一个 $AIRFLOW_HOME/webserver_config.py 文件来覆盖默认值。
如果要添加 webserver_config.py，它必须包含
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
喜欢 default webserver_config.py.
原则上，如果您在 airflow.cfg 中设置 rbac = True，删除 webserver_config.py 并执行任何命令，例如 airflow list_users 然后 airflow 将为创建一个 webserver_config.py 文件您使用适当的 SQLALCHEMY_DATABASE_URI = ... 可以在之后修改。