运行使用 IN 运算符进入一些 SQLite 限制

Question

我有一个使用 WHERE id IN (1,2,3,...) 的查询，其中列表 (1,2,3,...) 是从整数数组动态生成的（ 而不是 使用参数）。现在我有一个特定的查询，使用 26623 个 ID 大约需要 500 毫秒，但是使用 26624 个 ID 需要 50 秒（慢 100 倍）。

我在 https://sqlite.org/limits.html

中找不到相关内容

SELECT params.name AS name, json_group_array(DISTINCT params.value) AS "values"
FROM view_requests AS req, search_params(search) AS params
JOIN flows ON flows.request_id = req.id
WHERE search NOT IN ('', '?')
AND flows.id IN (1,2,3) /* <=== here more than 26623 IDs make it super slow  */
GROUP BY params.name
ORDER BY json_array_length("values") DESC, params.name ASC

在我尝试使它在隔离中可重现之前（例如 search_params 是一个自定义虚拟 table），有谁知道我可能运行受到什么限制？这不是 ID 本身的数量，因为不同的查询使用相同的 ID 运行得很好。

SQLite 版本 3.36.0 通过 better-sqlite3 (Node.js) 和只读数据库。我使用的唯一编译指示是 journal_mode = WAL.

编译为 (https://github.com/JoshuaWise/better-sqlite3/blob/master/docs/compilation.md#bundled-configuration):

SQLITE_DQS=0
SQLITE_LIKE_DOESNT_MATCH_BLOBS
SQLITE_THREADSAFE=2
SQLITE_USE_URI=0
SQLITE_DEFAULT_MEMSTATUS=0
SQLITE_OMIT_DEPRECATED
SQLITE_OMIT_GET_TABLE
SQLITE_OMIT_TCL_VARIABLE
SQLITE_OMIT_PROGRESS_CALLBACK
SQLITE_OMIT_SHARED_CACHE
SQLITE_TRACE_SIZE_LIMIT=32
SQLITE_DEFAULT_CACHE_SIZE=-16000
SQLITE_DEFAULT_FOREIGN_KEYS=1
SQLITE_DEFAULT_WAL_SYNCHRONOUS=1
SQLITE_ENABLE_MATH_FUNCTIONS
SQLITE_ENABLE_DESERIALIZE
SQLITE_ENABLE_COLUMN_METADATA
SQLITE_ENABLE_UPDATE_DELETE_LIMIT
SQLITE_ENABLE_STAT4
SQLITE_ENABLE_FTS3_PARENTHESIS
SQLITE_ENABLE_FTS3
SQLITE_ENABLE_FTS4
SQLITE_ENABLE_FTS5
SQLITE_ENABLE_JSON1
SQLITE_ENABLE_RTREE
SQLITE_ENABLE_GEOPOLY
SQLITE_INTROSPECTION_PRAGMAS
SQLITE_SOUNDEX
HAVE_STDINT_H=1
HAVE_INT8_T=1
HAVE_INT16_T=1
HAVE_INT32_T=1
HAVE_UINT8_T=1
HAVE_UINT16_T=1
HAVE_UINT32_T=1

Answer 1

这是来自 SQLite 论坛的答案。本质上，这是查询规划器如何处理 IN 文字和我的虚拟 table 估计成本的组合。这意味着我运行进入了查询规划器做出不同决定的确切时刻。

SQLite NGQP is a cost based query planner. The IN () operator with a list of literal values gets implemented as a kind of temporary table; sometimes SQLite decides to create an index and do lookups, other times it decides to use that table as the outermost loop of the query.

EXPLAIN QUERY PLAN should show that in a more concise manner.

If compiled in DEBUG mode mith WHERETRACE enabled, the .wheretrace command will show how SQLite NGQP reaches its plan. Essential input is the return values from the xBestIndex method of your virtual table, especially the "number of rows" and the "estimated cost". It is paramount to deliver accurate estimates. Cost should reflect processing cost relative to SQLite native tables.

Note that you can name the IN table by making it a CTE and CROSS JOIN to force the query plan that works fast.

https://sqlite.org/forum/forumpost/a3d68ed8b40cf583?t=h

我使用的解决方法是 json_each 并将整数数组序列化为 JSON 字符串。在我的特定用例中，这还有一些其他好处（例如，我可以绑定单个参数并使用任意数量的 ID 重新使用查询），所以我不介意这样做：

 SELECT params.name AS name, json_group_array(DISTINCT params.value) AS "values"
 FROM view_requests AS req, search_params(search) AS params
 JOIN flows ON flows.request_id = req.id
 WHERE search NOT IN ('', '?')
-AND flows.id IN (1,2,3)
+AND flows.id IN (SELECT value FROM json_each('[1,2,3]'))
 GROUP BY params.name
 ORDER BY json_array_length("values") DESC, params.name ASC

我也知道 better-sqlite3 的通用虚拟 table 实现在易于使用（简单得离谱）和实现最佳性能之间进行了权衡。

运行 使用 IN 运算符进入一些 SQLite 限制

Running into some SQLite limitation using IN operator

sqlite

better-sqlite3

运行使用 IN 运算符进入一些 SQLite 限制