Postgres 是否缓存我们的查询以及我们如何绕过它?
Is Postgres caching our queries and how do we get around it?
我正在尝试 运行 以下一段 python3 代码:
import os
import psycopg2
import logging
# Set max attempts before giving up
MAX_ATTEMPTS = 5
# Set basic logging config to debug (i.e. log everything).
# By default, this will log te stdout (i.e. it will behave the same as print)
logging.basicConfig(level=logging.DEBUG)
# Grab DB url from env variable
database_url = os.environ.get('DATABASE_URL')
assert database_url is not None, 'DATABASE_URL env variable must be set to a postgres connection string.'
# Initiate psycopg2 and instantiate a cursor object
conn = psycopg2.connect(database_url)
cursor = conn.cursor()
# Define function to delete old records
def delete_old_records(cur):
# execute a query to delete old records. We're going to refer to this as the "delete" command
query = 'DELETE FROM my_table WHERE id NOT IN ( SELECT id FROM ( SELECT id FROM my_table ORDER BY id DESC LIMIT 1850 ) foo);'
cur.execute(query)
# Set variables to keep track of loop
successful = False
attempts = 0
# While not successful and max attempts not reached
while not successful and attempts < MAX_ATTEMPTS:
try:
# Attempt to delete old records
delete_old_records(cursor)
# Set successful to True if no errors were encountered in the previous line
successful = True
# Log a message
logging.info('Successfully truncated old records!')
# If some psycopg2 error happens
except psycopg2.Error as e:
# Log the error
logging.exception('Got exception when executing query')
# Rollback the cursor and get ready to try again
conn.rollback()
# Increment attempts by 1
attempts += 1
# If not able to perform operation after max attempts, log message to indicate failure
if not successful:
logging.warning(f'Was not successfully able to truncate logs after {MAX_ATTEMPTS} retries. '
f'Check logs for traceback (console output by default).')
这是问题所在:
代码执行成功,没有错误。但是,当我们运行在postico(Postgres GUI for Mac)上执行以下命令(以下简称"count"命令)时:
SELECT count(*) from my_table;
我们得到 1860 而不是 1850(即行未被删除)。
当 运行在 psql 或 postico 中手动执行删除命令时,我们分别在 运行在 psql 或 postico 中执行 COUNT 命令时得到正确的结果。然而,当 运行 在 ipython.
中执行命令时,我们得到不同的结果
当我打开与计算机 A 上 ipython 上的数据库的连接时,我 运行 删除命令,然后打开另一个连接到 ipython 在计算机 B 上和 运行 计数命令,我看到数据库行数没有改变,即仍然是 1860,没有减少到 1850。
我怀疑 caching/memoization 但我不确定我的命令是否真的有效。 psycopg2、postico 或 postgres 本身是否有可能导致此问题的原因?我们如何解决这个问题?我们在 postico 或 psycopg2/postgres 上没有看到任何清晰的缓存。
不涉及缓存。 PostgreSQL 不缓存查询结果。
您只是忘记了 COMMIT
删除事务,因此它的影响在任何并发事务中都不可见。
我正在尝试 运行 以下一段 python3 代码:
import os
import psycopg2
import logging
# Set max attempts before giving up
MAX_ATTEMPTS = 5
# Set basic logging config to debug (i.e. log everything).
# By default, this will log te stdout (i.e. it will behave the same as print)
logging.basicConfig(level=logging.DEBUG)
# Grab DB url from env variable
database_url = os.environ.get('DATABASE_URL')
assert database_url is not None, 'DATABASE_URL env variable must be set to a postgres connection string.'
# Initiate psycopg2 and instantiate a cursor object
conn = psycopg2.connect(database_url)
cursor = conn.cursor()
# Define function to delete old records
def delete_old_records(cur):
# execute a query to delete old records. We're going to refer to this as the "delete" command
query = 'DELETE FROM my_table WHERE id NOT IN ( SELECT id FROM ( SELECT id FROM my_table ORDER BY id DESC LIMIT 1850 ) foo);'
cur.execute(query)
# Set variables to keep track of loop
successful = False
attempts = 0
# While not successful and max attempts not reached
while not successful and attempts < MAX_ATTEMPTS:
try:
# Attempt to delete old records
delete_old_records(cursor)
# Set successful to True if no errors were encountered in the previous line
successful = True
# Log a message
logging.info('Successfully truncated old records!')
# If some psycopg2 error happens
except psycopg2.Error as e:
# Log the error
logging.exception('Got exception when executing query')
# Rollback the cursor and get ready to try again
conn.rollback()
# Increment attempts by 1
attempts += 1
# If not able to perform operation after max attempts, log message to indicate failure
if not successful:
logging.warning(f'Was not successfully able to truncate logs after {MAX_ATTEMPTS} retries. '
f'Check logs for traceback (console output by default).')
这是问题所在:
代码执行成功,没有错误。但是,当我们运行在postico(Postgres GUI for Mac)上执行以下命令(以下简称"count"命令)时:
SELECT count(*) from my_table;
我们得到 1860 而不是 1850(即行未被删除)。
当 运行在 psql 或 postico 中手动执行删除命令时,我们分别在 运行在 psql 或 postico 中执行 COUNT 命令时得到正确的结果。然而,当 运行 在 ipython.
中执行命令时,我们得到不同的结果
当我打开与计算机 A 上 ipython 上的数据库的连接时,我 运行 删除命令,然后打开另一个连接到 ipython 在计算机 B 上和 运行 计数命令,我看到数据库行数没有改变,即仍然是 1860,没有减少到 1850。
我怀疑 caching/memoization 但我不确定我的命令是否真的有效。 psycopg2、postico 或 postgres 本身是否有可能导致此问题的原因?我们如何解决这个问题?我们在 postico 或 psycopg2/postgres 上没有看到任何清晰的缓存。
不涉及缓存。 PostgreSQL 不缓存查询结果。
您只是忘记了 COMMIT
删除事务,因此它的影响在任何并发事务中都不可见。