SQLALCHEMY_BINDS 的 flask-security 失去了静态文件池的连接

flask-security with SQLALCHEMY_BINDS loses connections from pool for static files

带有 SQLALCHEMY_BINDS 的 flask-security 失去了静态文件池的连接——至少我认为发生了什么。

Caching Flask-Login user_loader 下的第一条评论建议使用 flask-principal 的 skip_static=True,但我没有看到使用 flask-security 时可以访问 flask-principal 参数。

使用

Flask-Login==0.5.0
Flask-Mail==0.9.1
flask-nav==0.6
Flask-Principal==0.4.0
Flask-Security-Too==3.3.3
Flask-SQLAlchemy==2.4.1
Flask-WTF==0.14.3
SQLAlchemy==1.3.13

我初始化 SQLALCHEMY 变量如下

class RealDb(Config):
    def __init__(self, configfiles):
        if type(configfiles) == str:
            configfiles = [configfiles]

        # connect to database based on configuration
        config = {}
        for configfile in configfiles:
            config.update(getitems(configfile, 'database'))
        dbuser = config['dbuser']
        password = config['dbpassword']
        dbserver = config['dbserver']
        dbname = config['dbname']
        # app.logger.debug('using mysql://{uname}:*******@{server}/{dbname}'.format(uname=dbuser,server=dbserver,dbname=dbname))
        db_uri = 'mysql://{uname}:{pw}@{server}/{dbname}'.format(uname=dbuser, pw=password, server=dbserver,
                                                                 dbname=dbname)
        self.SQLALCHEMY_DATABASE_URI = db_uri
        # https://flask-sqlalchemy.palletsprojects.com/en/2.x/binds/
        userdbuser = config['userdbuser']
        userpassword = config['userdbpassword']
        userdbserver = config['userdbserver']
        userdbname = config['userdbname']
        userdb_uri = 'mysql://{uname}:{pw}@{server}/{dbname}'.format(uname=userdbuser, pw=userpassword, server=userdbserver,
                                                                 dbname=userdbname)
        self.SQLALCHEMY_BINDS = {
            'users': userdb_uri
        }

我的users模型(此时主模型是空的)

# pypi
from flask_sqlalchemy import SQLAlchemy
from flask_security import UserMixin, RoleMixin

# set up database - SQLAlchemy() must be done after app.config SQLALCHEMY_* assignments
db = SQLAlchemy()
Table = db.Table
Column = db.Column
Integer = db.Integer
Float = db.Float
Boolean = db.Boolean
String = db.String
Text = db.Text
Date = db.Date
Time = db.Time
DateTime = db.DateTime
Sequence = db.Sequence
Enum = db.Enum
UniqueConstraint = db.UniqueConstraint
ForeignKey = db.ForeignKey
relationship = db.relationship
backref = db.backref
object_mapper = db.object_mapper
Base = db.Model

# some string sizes
DESCR_LEN = 512
INTEREST_LEN = 32
APPLICATION_LEN = 32
# role management, some of these are overloaded
USERROLEDESCR_LEN = 512
ROLENAME_LEN = 32
EMAIL_LEN = 100
NAME_LEN = 256
PASSWORD_LEN = 255
UNIQUIFIER_LEN = 255

# common roles
ROLE_SUPER_ADMIN = 'super-admin'

userinterest_table = Table('users_interests', Base.metadata,
                           Column('user_id', Integer, ForeignKey('user.id')),
                           Column('interest_id', Integer, ForeignKey('interest.id')),
                           info={'bind_key': 'users'},
                          )

appinterest_table = Table('apps_interests', Base.metadata,
                           Column('application_id', Integer, ForeignKey('application.id')),
                           Column('interest_id', Integer, ForeignKey('interest.id')),
                           info={'bind_key': 'users'},
                          )

class Interest(Base):
    __tablename__ = 'interest'
    __bind_key__ = 'users'
    id                  = Column(Integer(), primary_key=True)
    version_id          = Column(Integer, nullable=False, default=1)
    interest            = Column(String(INTEREST_LEN))
    users               = relationship("User",
                                       secondary=userinterest_table,
                                       backref=backref("interests"))
    applications        = relationship("Application",
                                       secondary=appinterest_table,
                                       backref=backref("interests"))
    description         = Column(String(DESCR_LEN))
    public              = Column(Boolean)

class Application(Base):
    __tablename__ = 'application'
    __bind_key__ = 'users'
    id              = Column(Integer(), primary_key=True)
    application     = Column(String(APPLICATION_LEN))

# user role management
# adapted from
#   https://flask-security-too.readthedocs.io/en/stable/quickstart.html (SQLAlchemy Application)

class RolesUsers(Base):
    __tablename__ = 'roles_users'
    __bind_key__ = 'users'
    id = Column(Integer(), primary_key=True)
    user_id = Column('user_id', Integer(), ForeignKey('user.id'))
    role_id = Column('role_id', Integer(), ForeignKey('role.id'))

class Role(Base, RoleMixin):
    __tablename__ = 'role'
    __bind_key__ = 'users'
    id                  = Column(Integer(), primary_key=True)
    version_id          = Column(Integer, nullable=False, default=1)
    name                = Column(String(ROLENAME_LEN), unique=True)
    description         = Column(String(USERROLEDESCR_LEN))

class User(Base, UserMixin):
    __tablename__ = 'user'
    __bind_key__ = 'users'
    id                  = Column(Integer, primary_key=True)
    version_id          = Column(Integer, nullable=False, default=1)
    email               = Column( String(EMAIL_LEN), unique=True )  # = username
    password            = Column( String(PASSWORD_LEN) )
    name                = Column( String(NAME_LEN) )
    given_name          = Column( String(NAME_LEN) )
    last_login_at       = Column( DateTime() )
    current_login_at    = Column( DateTime() )
    last_login_ip       = Column( String(100) )
    current_login_ip    = Column( String(100) )
    login_count         = Column( Integer )
    active              = Column( Boolean() )
    fs_uniquifier       = Column( String(UNIQUIFIER_LEN) )
    confirmed_at        = Column( DateTime() )
    roles               = relationship('Role', secondary='roles_users',
                          backref=backref('users', lazy='dynamic'))

记录连接池:

logging.basicConfig()
logging.getLogger('sqlalchemy.pool').setLevel(logging.DEBUG)

使用上面的方法我看到静态文件的连接未释放,最终导致 sqlalchemy.exc.TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)。 (我相信 /admin/users 中的 db.session.commit() 会导致返回该连接)。

DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 0000020389174368>
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 0000020389174368> checked out from pool
127.0.0.1 - - [10/Mar/2020 15:10:03] "GET /static/js/jquery-ui-1.12.1.custom/images/ui-bg_glass_50_3baae3_1x400.png HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [10/Mar/2020 15:10:03] "GET /static/js/jquery-ui-1.12.1.custom/images/ui-bg_glass_50_3baae3_1x400.png HTTP/1.1" 200 -
DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 00000203891761C8>
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 00000203891761C8> checked out from pool
request.path = /admin/users
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 00000203891761C8> being returned to pool
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 00000203891761C8> rollback-on-return
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 00000203891761C8> checked out from pool
[2020-03-10 15:10:05,773] DEBUG in tables: rendertemplate(): self.templateargs = {}
DEBUG:members:rendertemplate(): self.templateargs = {}
127.0.0.1 - - [10/Mar/2020 15:10:05] "GET /admin/users HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [10/Mar/2020 15:10:05] "GET /admin/users HTTP/1.1" 200 -
DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 0000020389175298>
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 0000020389175298> checked out from pool
127.0.0.1 - - [10/Mar/2020 15:10:06] "GET /static/js/jquery-ui-1.12.1.custom/jquery-ui.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [10/Mar/2020 15:10:06] "GET /static/js/jquery-ui-1.12.1.custom/jquery-ui.css HTTP/1.1" 200 -
DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 00000203891770F8>
                                       :
Traceback (most recent call last):
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 1947, in full_dispatch_request
    rv = self.preprocess_request()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask\app.py", line 2241, in preprocess_request
    rv = func()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_principal.py", line 477, in _on_before_request
    identity = loader()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_security\core.py", line 405, in _identity_loader
    if not isinstance(current_user._get_current_object(), AnonymousUserMixin):
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\werkzeug\local.py", line 306, in _get_current_object
    return self.__local()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_login\utils.py", line 26, in <lambda>
    current_user = LocalProxy(lambda: _get_user())
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_login\utils.py", line 346, in _get_user
    current_app.login_manager._load_user()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_login\login_manager.py", line 318, in _load_user
    user = self._user_callback(user_id)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_security\core.py", line 345, in _user_loader
    user = _security.datastore.find_user(id=user_id)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\flask_security\datastore.py", line 382, in find_user
    return query.filter_by(**kwargs).first()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3287, in first
    ret = list(self[0:1])
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3065, in __getitem__
    return list(res)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3389, in __iter__
    return self._execute_and_instances(context)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3411, in _execute_and_instances
    querycontext, self._connection_from_session, close_with_result=True
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3426, in _get_bind_args
    mapper=self._bind_mapper(), clause=querycontext.statement, **kw
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3404, in _connection_from_session
    conn = self.session.connection(**kw)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\session.py", line 1133, in connection
    execution_options=execution_options,
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\session.py", line 1139, in _connection_for_bind
    engine, execution_options
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\orm\session.py", line 432, in _connection_for_bind
    conn = bind._contextual_connect()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2242, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2276, in _wrap_pool_connect
    return fn()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\pool\base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\pool\base.py", line 773, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\pool\base.py", line 492, in checkout
    rec = pool._do_get()
  File "C:\Users\lking\Documents\Lou's Software\projects\members\members\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 131, in _do_get
    code="3o7r",
sqlalchemy.exc.TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)

如果我不使用 SQLALCHEMY_BINDS 连接将释放到池中,类似于以下内容:

DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8>
DEBUG:sqlalchemy.pool.impl.QueuePool:Created new connection <_mysql.connection open to '127.0.0.1' at 000002A68ED94F88>
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> checked out from pool
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED94F88> checked out from pool
request.path = /admin/users
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> being returned to pool
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> rollback-on-return, via agent
127.0.0.1 - - [10/Mar/2020 15:25:44] "GET /static/js/jquery-ui-1.12.1.custom/images/ui-icons_ffffff_256x240.png HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [10/Mar/2020 15:25:44] "GET /static/js/jquery-ui-1.12.1.custom/images/ui-icons_ffffff_256x240.png HTTP/1.1" 200 -
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED94F88> being returned to pool
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED94F88> rollback-on-return
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> checked out from pool
[2020-03-10 15:25:44,829] DEBUG in tables: rendertemplate(): self.templateargs = {}
DEBUG:runningroutes:rendertemplate(): self.templateargs = {}
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> being returned to pool
DEBUG:sqlalchemy.pool.impl.QueuePool:Connection <_mysql.connection open to '127.0.0.1' at 000002A68ED917D8> rollback-on-return, via agent
         :

问题是我使用了两个不同的 SQLAlchemy() 实例。不知道我怎么会看这个两天,在发布到 Whosebug 后五分钟内我顿悟了。