为什么我的 Python Azure App Service 在部署一段时间后会导致 "sqlalchemy.exc.InvalidRequestError"?

Why does my Python Azure App Service result in "sqlalchemy.exc.InvalidRequestError" after it has been deployed for a while?

我有一个使用 Python、Flask 和 Flask-Security-Too 的网络应用程序。当 运行 在本地针对部署的数据库(也在 Azure 中)时,它按预期工作。当应用程序部署到 Azure 时,它​​也能很好地工作。但是,一段时间后它会开始抛出如下错误:

2021-11-25T08:33:20.676633302Z [2021-11-25 08:33:20 +0000] [8] [ERROR] Error handling request /login
2021-11-25T08:33:20.676686905Z Traceback (most recent call last):
2021-11-25T08:33:20.676694906Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1202, in _execute_context
2021-11-25T08:33:20.676700406Z     conn = self._revalidate_connection()
2021-11-25T08:33:20.676705406Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 469, in _revalidate_connection
2021-11-25T08:33:20.676710407Z     raise exc.InvalidRequestError(
2021-11-25T08:33:20.676715107Z sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
2021-11-25T08:33:20.676720007Z 
2021-11-25T08:33:20.676724408Z The above exception was the direct cause of the following exception:
2021-11-25T08:33:20.676729108Z 
2021-11-25T08:33:20.676733508Z Traceback (most recent call last):
2021-11-25T08:33:20.676738108Z   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
2021-11-25T08:33:20.676742809Z     self.handle_request(listener, req, client, addr)
2021-11-25T08:33:20.676747309Z   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
2021-11-25T08:33:20.676752109Z     respiter = self.wsgi(environ, resp.start_response)
2021-11-25T08:33:20.676756710Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
2021-11-25T08:33:20.676761610Z     return self.wsgi_app(environ, start_response)
2021-11-25T08:33:20.676766110Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
2021-11-25T08:33:20.676770711Z     response = self.handle_exception(e)
2021-11-25T08:33:20.676775311Z   File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
2021-11-25T08:33:20.676780011Z     return cors_after_request(app.make_response(f(*args, **kwargs)))
2021-11-25T08:33:20.676784712Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
2021-11-25T08:33:20.676789412Z     reraise(exc_type, exc_value, tb)
2021-11-25T08:33:20.676794012Z   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
2021-11-25T08:33:20.676798612Z     raise value
2021-11-25T08:33:20.676813213Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
2021-11-25T08:33:20.676818114Z     response = self.full_dispatch_request()
2021-11-25T08:33:20.676822214Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
2021-11-25T08:33:20.676826514Z     rv = self.handle_user_exception(e)
2021-11-25T08:33:20.676830615Z   File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
2021-11-25T08:33:20.676834915Z     return cors_after_request(app.make_response(f(*args, **kwargs)))
2021-11-25T08:33:20.676840015Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
2021-11-25T08:33:20.676844515Z     reraise(exc_type, exc_value, tb)
2021-11-25T08:33:20.676848616Z   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
2021-11-25T08:33:20.676852916Z     raise value
2021-11-25T08:33:20.676874217Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
2021-11-25T08:33:20.676878918Z     rv = self.dispatch_request()
2021-11-25T08:33:20.676883218Z   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
2021-11-25T08:33:20.676888318Z     return self.view_functions[rule.endpoint](**req.view_args)
2021-11-25T08:33:20.676892919Z   File "/usr/local/lib/python3.8/site-packages/flask_security/decorators.py", line 436, in decorated
2021-11-25T08:33:20.676897419Z     return fn(*args, **kwargs)
2021-11-25T08:33:20.676901819Z   File "/usr/local/lib/python3.8/site-packages/flask_security/views.py", line 162, in login
2021-11-25T08:33:20.676906220Z     if form.validate_on_submit():
2021-11-25T08:33:20.676910620Z   File "/usr/local/lib/python3.8/site-packages/flask_wtf/form.py", line 100, in validate_on_submit
2021-11-25T08:33:20.676915220Z     return self.is_submitted() and self.validate()
2021-11-25T08:33:20.676919720Z   File "/usr/local/lib/python3.8/site-packages/flask_security/forms.py", line 450, in validate
2021-11-25T08:33:20.676924121Z     self.user = find_user(self.email.data)
2021-11-25T08:33:20.676928321Z   File "/usr/local/lib/python3.8/site-packages/flask_security/utils.py", line 829, in find_user
2021-11-25T08:33:20.676932921Z     user = _datastore.find_user(
2021-11-25T08:33:20.676937122Z   File "/usr/local/lib/python3.8/site-packages/flask_security/datastore.py", line 585, in find_user
2021-11-25T08:33:20.676941622Z     return query.filter(subquery).first()
2021-11-25T08:33:20.676945822Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3429, in first
2021-11-25T08:33:20.676950422Z     ret = list(self[0:1])
2021-11-25T08:33:20.676954623Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
2021-11-25T08:33:20.676959023Z     return list(res)
2021-11-25T08:33:20.676967724Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
2021-11-25T08:33:20.676972424Z     return self._execute_and_instances(context)
2021-11-25T08:33:20.676976724Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
2021-11-25T08:33:20.676981224Z     result = conn.execute(querycontext.statement, self._params)
2021-11-25T08:33:20.676985525Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
2021-11-25T08:33:20.676989925Z     return meth(self, multiparams, params)
2021-11-25T08:33:20.676994525Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2021-11-25T08:33:20.676999126Z     return connection._execute_clauseelement(self, multiparams, params)
2021-11-25T08:33:20.677003426Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
2021-11-25T08:33:20.677007926Z     ret = self._execute_context(
2021-11-25T08:33:20.677012127Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1206, in _execute_context
2021-11-25T08:33:20.677016627Z     self._handle_dbapi_exception(
2021-11-25T08:33:20.677020827Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
2021-11-25T08:33:20.677025327Z     util.raise_(
2021-11-25T08:33:20.677029528Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2021-11-25T08:33:20.677034028Z     raise exception
2021-11-25T08:33:20.677038128Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1202, in _execute_context
2021-11-25T08:33:20.677042729Z     conn = self._revalidate_connection()
2021-11-25T08:33:20.677046929Z   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 469, in _revalidate_connection
2021-11-25T08:33:20.677051429Z     raise exc.InvalidRequestError(
2021-11-25T08:33:20.677055629Z sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
2021-11-25T08:33:20.677061730Z [SQL: SELECT anon_1.user_id AS anon_1_user_id, anon_1.user_email AS anon_1_user_email, anon_1.user_username AS anon_1_user_username, anon_1.user_password AS anon_1_user_password, anon_1.user_active AS anon_1_user_active, anon_1.user_fs_uniquifier AS anon_1_user_fs_uniquifier, anon_1.user_confirmed_at AS anon_1_user_confirmed_at, anon_1.user_last_login_at AS anon_1_user_last_login_at, anon_1.user_current_login_at AS anon_1_user_current_login_at, anon_1.user_last_login_ip AS anon_1_user_last_login_ip, anon_1.user_current_login_ip AS anon_1_user_current_login_ip, anon_1.user_login_count AS anon_1_user_login_count, anon_1.user_tf_primary_method AS anon_1_user_tf_primary_method, anon_1.user_tf_totp_secret AS anon_1_user_tf_totp_secret, anon_1.user_tf_phone_number AS anon_1_user_tf_phone_number, anon_1.user_create_datetime AS anon_1_user_create_datetime, anon_1.user_update_datetime AS anon_1_user_update_datetime, anon_1.user_uuid AS anon_1_user_uuid, role_1.id AS role_1_id, role_1.name AS role_1_name, role_1.description AS role_1_description, role_1.permissions AS role_1_permissions, role_1.update_datetime AS role_1_update_datetime 
2021-11-25T08:33:20.677072630Z FROM (SELECT TOP 1 [user].id AS user_id, [user].email AS user_email, [user].username AS user_username, [user].password AS user_password, [user].active AS user_active, [user].fs_uniquifier AS user_fs_uniquifier, [user].confirmed_at AS user_confirmed_at, [user].last_login_at AS user_last_login_at, [user].current_login_at AS user_current_login_at, [user].last_login_ip AS user_last_login_ip, [user].current_login_ip AS user_current_login_ip, [user].login_count AS user_login_count, [user].tf_primary_method AS user_tf_primary_method, [user].tf_totp_secret AS user_tf_totp_secret, [user].tf_phone_number AS user_tf_phone_number, [user].create_datetime AS user_create_datetime, [user].update_datetime AS user_update_datetime, [user].uuid AS user_uuid 
2021-11-25T08:33:20.677078531Z FROM [user] 
2021-11-25T08:33:20.677082731Z WHERE lower([user].email) = lower(?)) AS anon_1 LEFT OUTER JOIN (roles_users AS roles_users_1 JOIN role AS role_1 ON role_1.id = roles_users_1.role_id) ON anon_1.user_id = roles_users_1.user_id]
2021-11-25T08:33:20.677087431Z [parameters: [immutabledict({})]]

AFAIK,所有 db.sessionuser_datastore 实体要么回滚,要么在使用后提交 and/or 关闭。这是一个例子:

def create_user():
    try:
        current_app.logger.info('Creating admin user...')
        setup_admin_user(current_app.user_datastore)
        current_app.logger.info('Done!')
    except IntegrityError as ie:
        current_app.logger.warning('Failed to create admin user(s) as they already exists.')
        db.session.rollback()
    except Exception as e:
        db.session.rollback()
        raise e
    finally:
        db.session.close()


def setup_admin_user(usr_datastore):
    if not usr_datastore.find_user(email="email@example.com"):
        usr_datastore.create_user(email="email@example.com",
                                  password=hash_password('password'),
                                  roles=["admin"],
                                  confirmed_at=datetime.datetime.utcnow())
        usr_datastore.commit()

我还注意到所提供的 SQL 的参数看起来有点可疑。有时有一个真正的参数,但错误信息仍然是一样的。

我很困惑,因为查询在数据库中执行时确实工作正常,并且 Traceback 没有提到我编写的代码中的任何行。

知道如何解决这个问题吗?我已经尝试将 flask-sqlalchemyflask-security-too 的版本升级到最新的稳定版本。

这有时会发生,因为云数据库实例通常会关闭空闲连接。确保您的配置中包含以下内容:

app.config["SQLALCHEMY_ENGINE_OPTIONS"] = {"pool_pre_ping": 正确}

看看是否有帮助。

通过为 Azure 中的应用服务配置健康检查解决了这个问题。健康检查端点将对数据库进行简单查询以使其保持活动状态,我认为这是问题所在。

def health_check():
    role = current_app.user_datastore.find_role('dummy')
    if role:
        current_app.logger.info(f'[health_check] Found role {role.name}')
        return jsonify(status='ok'), 200
    return jsonify(status='failure'), 500