postgres LockError ...如何调查

Question

您好，我将 gunicorn 与 nginx 和 postgreSQL 数据库一起用于运行我的网络应用程序。我最近更改了

中的 gunicorn 命令

gunicorn run:app -w 4 -b 0.0.0.0:8080 --workers=1 --timeout=300

到

gunicorn run:app -w 4 -b 0.0.0.0:8080 --workers=2 --timeout=300

使用 2 个工人。现在我收到类似

的错误消息

  File "/usr/local/lib/python2.7/dist-packages/flask_sqlalchemy/__init__.py", line 194, in session_signal_after_commit
    models_committed.send(session.app, changes=list(d.values()))
  File "/usr/local/lib/python2.7/dist-packages/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python2.7/dist-packages/flask_whooshalchemy.py", line 265, in _after_flush
    with index.writer() as writer:
  File "/usr/local/lib/python2.7/dist-packages/whoosh/index.py", line 464, in writer
    return SegmentWriter(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/whoosh/writing.py", line 502, in __init__
    raise LockError
LockError

我真的不能对这些错误消息做太多事情，但它们似乎与我在数据库模型table 中对用户 table 进行的快速搜索相关联

import sys
if sys.version_info >= (3, 0):
    enable_search = False
else:
    enable_search = True
    import flask.ext.whooshalchemy as whooshalchemy

class User(db.Model):
    __searchable__ = ['username','email','position','institute','id'] # these fields will be indexed by whoosh

    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(100), index=True)
    ...

    def __repr__(self):
        return '<User %r>' % (self.username)

if enable_search:
    whooshalchemy.whoosh_index(app, User)

有什么想法可以调查吗？我认为 postgres 允许并行访问，因此我认为不应该发生锁定错误？当我只使用 1 个工作时，它们没有发生，所以这肯定是由于有多个工作人员造成的...... 任何帮助表示赞赏谢谢卡尔

Answer 1

这与 PostgreSQL 无关。 Whoosh 持有用于写入的文件锁，但在此代码的最后一行失败...

class SegmentWriter(IndexWriter):
    def __init__(self, ix, poolclass=None, timeout=0.0, delay=0.1, _lk=True,
                 limitmb=128, docbase=0, codec=None, compound=True, **kwargs):
        # Lock the index
        self.writelock = None 
        if _lk: 
            self.writelock = ix.lock("WRITELOCK")
            if not try_for(self.writelock.acquire, timeout=timeout,
                           delay=delay):
                raise LockError

请注意，延迟默认为 0.1 秒，如果在此时间内未获得锁定，它将失败。你增加了你的工人，所以现在你有锁的争用。来自以下文档...

https://whoosh.readthedocs.org/en/latest/threads.html

Locking

Only one thread/process can write to an index at a time. When you open a writer, it locks the index. If you try to open a writer on the same index in another thread/process, it will raise whoosh.store.LockError.

In a multi-threaded or multi-process environment your code needs to be aware that opening a writer may raise this exception if a writer is already open. Whoosh includes a couple of example implementations (whoosh.writing.AsyncWriter and whoosh.writing.BufferedWriter) of ways to work around the write lock.

While the writer is open and during the commit, the index is still available for reading. Existing readers are unaffected and new readers can open the current index normally.

您可以找到有关如何同时使用 Whoosh 的示例。

缓冲

https://whoosh.readthedocs.org/en/latest/api/writing.html#whoosh.writing.BufferedWriter

异步

https://whoosh.readthedocs.org/en/latest/api/writing.html#whoosh.writing.AsyncWriter

我会先尝试缓冲版本，因为批处理写入几乎总是更快。

postgres LockError ...如何调查

postgres LockError... how to investigate

locking

whoosh