我如何让 postgres 避免为此搜索分页查询执行双顺序扫描？
How do I make postgres avoid doing a double sequential scan for this seek pagination query?
架构
- 我有一堆 post 存储在 table (feed_items)
- 我有一个 table 包含哪个用户 ID liked/disliked 哪个 feed_item_id (feed_item_likes_dislikes)
- 我有另一个 table 包含哪个用户 ID loved/angered 哪个 feed_item_id (feed_item_love_anger)
- 我有第四个 table 包含其中 feed_item_id 有什么标签 其中标签是一个 ARRAY of varchar (feed_item_tags)
- 每 post 的总数 likes/dislikes 存储在实体化视图中 (feed_item_likes_dislikes_aggregate)
- 总数love/anger存储在另一个物化视图(feed_item_love_anger_agregate)
- 喜厌爱怒分开存放，因为一个post可以同时是liked/disliked和loved/angered（可惜业务需求）
- 我在 feed_items 中有 2 个名为 title_vector 和 summary_vector 的列，类型为 TSVECTOR，这有助于通过搜索关键字查找 posts（在 postgres)
问题
- 我想按发布日期降序查找所有 post 和 feed_item_id
- 一些post同时发布，我想使用(pubdate, feed_item_id) < (value1, value2) seek HERE[=中描述的分页方法进行分页58=]
我的第 1 页查询
找到 post 点赞数 > 0 的标题或摘要中标记为
的单词 scam
SELECT
fi.feed_item_id,
pubdate,
link,
title,
summary,
author,
feed_id,
likes,
dislikes,
love,
anger,
tags
FROM
feed_items fi
LEFT JOIN
feed_item_tags t
ON fi.feed_item_id = t.feed_item_id
LEFT JOIN
feed_item_love_anger_aggregate bba
ON fi.feed_item_id = bba.feed_item_id
LEFT JOIN
feed_item_likes_dislikes_aggregate lda
ON fi.feed_item_id = lda.feed_item_id
WHERE
(
title_vector @@ to_tsquery('scam')
OR summary_vector @@ to_tsquery('scam')
)
AND 'for' = ANY(tags)
AND likes > 0
ORDER BY
pubdate DESC,
feed_item_id DESC LIMIT 3;
解释分析第 1 页
Limit (cost=2.83..16.88 rows=3 width=233) (actual time=0.075..0.158 rows=3 loops=1)
-> Nested Loop Left Join (cost=2.83..124.53 rows=26 width=233) (actual time=0.074..0.157 rows=3 loops=1)
-> Nested Loop (cost=2.69..116.00 rows=26 width=217) (actual time=0.067..0.146 rows=3 loops=1)
Join Filter: (t.feed_item_id = fi.feed_item_id)
Rows Removed by Join Filter: 73
-> Index Scan using idx_feed_items_pubdate_feed_item_id_desc on feed_items fi (cost=0.14..68.77 rows=76 width=62) (actual time=0.016..0.023 rows=3 loops=1)
Filter: ((title_vector @@ to_tsquery('scam'::text)) OR (summary_vector @@ to_tsquery('scam'::text)))
Rows Removed by Filter: 1
-> Materialize (cost=2.55..8.56 rows=34 width=187) (actual time=0.016..0.037 rows=25 loops=3)
-> Hash Join (cost=2.55..8.39 rows=34 width=187) (actual time=0.044..0.091 rows=36 loops=1)
Hash Cond: (t.feed_item_id = lda.feed_item_id)
-> Seq Scan on feed_item_tags t (cost=0.00..5.25 rows=67 width=155) (actual time=0.009..0.043 rows=67 loops=1)
Filter: ('for'::text = ANY ((tags)::text[]))
Rows Removed by Filter: 33
-> Hash (cost=1.93..1.93 rows=50 width=32) (actual time=0.029..0.029 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on feed_item_likes_dislikes_aggregate lda (cost=0.00..1.93 rows=50 width=32) (actual time=0.004..0.013 rows=50 loops=1)
Filter: (likes > 0)
Rows Removed by Filter: 24
-> Index Scan using idx_feed_item_love_anger_aggregate on feed_item_love_anger_aggregate bba (cost=0.14..0.32 rows=1 width=32) (actual time=0.002..0.003 rows=0 loops=3)
Index Cond: (feed_item_id = fi.feed_item_id)
Planning Time: 0.601 ms
Execution Time: 0.195 ms
(23 rows)
尽管在所有 tables
上都有适当的索引，但它仍在进行 2 次顺序扫描
我的第N页查询
从上述查询中获取第 3 个结果的发布日期和 feed_item_id 并加载接下来的 3 个结果
SELECT
fi.feed_item_id,
pubdate,
link,
title,
summary,
author,
feed_id,
likes,
dislikes,
love,
anger,
tags
FROM
feed_items fi
LEFT JOIN
feed_item_tags t
ON fi.feed_item_id = t.feed_item_id
LEFT JOIN
feed_item_love_anger_aggregate bba
ON fi.feed_item_id = bba.feed_item_id
LEFT JOIN
feed_item_likes_dislikes_aggregate lda
ON fi.feed_item_id = lda.feed_item_id
WHERE
(
pubdate,
fi.feed_item_id
)
< ('2020-06-19 19:50:00+05:30', 'bc5c8dfe-13a9-d97a-a328-0e5b8990c500')
AND
(
title_vector @@ to_tsquery('scam')
OR summary_vector @@ to_tsquery('scam')
)
AND 'for' = ANY(tags)
AND likes > 0
ORDER BY
pubdate DESC,
feed_item_id DESC LIMIT 3;
解释第 N 页查询
尽管进行了过滤，但它仍在进行 2 次顺序扫描
Limit (cost=2.83..17.13 rows=3 width=233) (actual time=0.082..0.199 rows=3 loops=1)
-> Nested Loop Left Join (cost=2.83..121.97 rows=25 width=233) (actual time=0.081..0.198 rows=3 loops=1)
-> Nested Loop (cost=2.69..113.67 rows=25 width=217) (actual time=0.073..0.185 rows=3 loops=1)
Join Filter: (t.feed_item_id = fi.feed_item_id)
Rows Removed by Join Filter: 183
-> Index Scan using idx_feed_items_pubdate_feed_item_id_desc on feed_items fi (cost=0.14..67.45 rows=74 width=62) (actual time=0.014..0.034 rows=6 loops=1)
Index Cond: (ROW(pubdate, feed_item_id) < ROW('2020-06-19 19:50:00+05:30'::timestamp with time zone, 'bc5c8dfe-13a9-d97a-a328-0e5b8990c500'::uuid))
Filter: ((title_vector @@ to_tsquery('scam'::text)) OR (summary_vector @@ to_tsquery('scam'::text)))
Rows Removed by Filter: 2
-> Materialize (cost=2.55..8.56 rows=34 width=187) (actual time=0.009..0.022 rows=31 loops=6)
-> Hash Join (cost=2.55..8.39 rows=34 width=187) (actual time=0.050..0.098 rows=36 loops=1)
Hash Cond: (t.feed_item_id = lda.feed_item_id)
-> Seq Scan on feed_item_tags t (cost=0.00..5.25 rows=67 width=155) (actual time=0.009..0.044 rows=67 loops=1)
Filter: ('for'::text = ANY ((tags)::text[]))
Rows Removed by Filter: 33
-> Hash (cost=1.93..1.93 rows=50 width=32) (actual time=0.028..0.029 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on feed_item_likes_dislikes_aggregate lda (cost=0.00..1.93 rows=50 width=32) (actual time=0.005..0.014 rows=50 loops=1)
Filter: (likes > 0)
Rows Removed by Filter: 24
-> Index Scan using idx_feed_item_love_anger_aggregate on feed_item_love_anger_aggregate bba (cost=0.14..0.32 rows=1 width=32) (actual time=0.003..0.003 rows=1 loops=3)
Index Cond: (feed_item_id = fi.feed_item_id)
Planning Time: 0.596 ms
Execution Time: 0.236 ms
(24 rows)
我已经用所需的 table 和索引设置了 fiddle，有人可以告诉我如何修复查询以最好或减少使用索引扫描顺序扫描的数量为 1?
结构'for' = ANY(tags)不能使用GIN索引。为了能够使用它，您需要将其重新表述为 '{for}' <@ tags.
然而，它会选择不使用索引，因为 table 太小，条件太无选择性。如果你想强制使用索引，为了证明它能够这样做，你可以先 set enable_seqscan=off.
除了 GIN 索引，您目前在标签 table 上没有其他索引。在你的 fiddle 中，如果我 create index on feed_item_tags (feed_item_id) 并执行 ANALYZE，那么两个序列扫描都会消失。这样做可能比重新制定更好，以便它可以使用 GIN 索引，就像我的其他答案一样，因为这种方式可以更有效地利用 LIMIT 提前停止的前景。
但实际上，“feed_item_tags”table 的意义何在？如果您要使用 child table 来列出标签，通常每行会有一个 tag/parent_id 组合。如果您想要一个标签数组而不是一列标签，为什么不直接将数组粘贴到 parent table 中呢？有时 table 与两个 table 之间的 1:1 关系是有原因的，但并不常见。
架构
- 我有一堆 post 存储在 table (feed_items)
- 我有一个 table 包含哪个用户 ID liked/disliked 哪个 feed_item_id (feed_item_likes_dislikes)
- 我有另一个 table 包含哪个用户 ID loved/angered 哪个 feed_item_id (feed_item_love_anger)
- 我有第四个 table 包含其中 feed_item_id 有什么标签 其中标签是一个 ARRAY of varchar (feed_item_tags)
- 每 post 的总数 likes/dislikes 存储在实体化视图中 (feed_item_likes_dislikes_aggregate)
- 总数love/anger存储在另一个物化视图(feed_item_love_anger_agregate)
- 喜厌爱怒分开存放，因为一个post可以同时是liked/disliked和loved/angered（可惜业务需求）
- 我在 feed_items 中有 2 个名为 title_vector 和 summary_vector 的列，类型为 TSVECTOR，这有助于通过搜索关键字查找 posts（在 postgres)
问题
- 我想按发布日期降序查找所有 post 和 feed_item_id
- 一些post同时发布，我想使用(pubdate, feed_item_id) < (value1, value2) seek HERE[=中描述的分页方法进行分页58=]
我的第 1 页查询
找到 post 点赞数 > 0 的标题或摘要中标记为
的单词 scamSELECT
fi.feed_item_id,
pubdate,
link,
title,
summary,
author,
feed_id,
likes,
dislikes,
love,
anger,
tags
FROM
feed_items fi
LEFT JOIN
feed_item_tags t
ON fi.feed_item_id = t.feed_item_id
LEFT JOIN
feed_item_love_anger_aggregate bba
ON fi.feed_item_id = bba.feed_item_id
LEFT JOIN
feed_item_likes_dislikes_aggregate lda
ON fi.feed_item_id = lda.feed_item_id
WHERE
(
title_vector @@ to_tsquery('scam')
OR summary_vector @@ to_tsquery('scam')
)
AND 'for' = ANY(tags)
AND likes > 0
ORDER BY
pubdate DESC,
feed_item_id DESC LIMIT 3;
解释分析第 1 页
Limit (cost=2.83..16.88 rows=3 width=233) (actual time=0.075..0.158 rows=3 loops=1)
-> Nested Loop Left Join (cost=2.83..124.53 rows=26 width=233) (actual time=0.074..0.157 rows=3 loops=1)
-> Nested Loop (cost=2.69..116.00 rows=26 width=217) (actual time=0.067..0.146 rows=3 loops=1)
Join Filter: (t.feed_item_id = fi.feed_item_id)
Rows Removed by Join Filter: 73
-> Index Scan using idx_feed_items_pubdate_feed_item_id_desc on feed_items fi (cost=0.14..68.77 rows=76 width=62) (actual time=0.016..0.023 rows=3 loops=1)
Filter: ((title_vector @@ to_tsquery('scam'::text)) OR (summary_vector @@ to_tsquery('scam'::text)))
Rows Removed by Filter: 1
-> Materialize (cost=2.55..8.56 rows=34 width=187) (actual time=0.016..0.037 rows=25 loops=3)
-> Hash Join (cost=2.55..8.39 rows=34 width=187) (actual time=0.044..0.091 rows=36 loops=1)
Hash Cond: (t.feed_item_id = lda.feed_item_id)
-> Seq Scan on feed_item_tags t (cost=0.00..5.25 rows=67 width=155) (actual time=0.009..0.043 rows=67 loops=1)
Filter: ('for'::text = ANY ((tags)::text[]))
Rows Removed by Filter: 33
-> Hash (cost=1.93..1.93 rows=50 width=32) (actual time=0.029..0.029 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on feed_item_likes_dislikes_aggregate lda (cost=0.00..1.93 rows=50 width=32) (actual time=0.004..0.013 rows=50 loops=1)
Filter: (likes > 0)
Rows Removed by Filter: 24
-> Index Scan using idx_feed_item_love_anger_aggregate on feed_item_love_anger_aggregate bba (cost=0.14..0.32 rows=1 width=32) (actual time=0.002..0.003 rows=0 loops=3)
Index Cond: (feed_item_id = fi.feed_item_id)
Planning Time: 0.601 ms
Execution Time: 0.195 ms
(23 rows)
尽管在所有 tables
上都有适当的索引，但它仍在进行 2 次顺序扫描我的第N页查询
从上述查询中获取第 3 个结果的发布日期和 feed_item_id 并加载接下来的 3 个结果
SELECT
fi.feed_item_id,
pubdate,
link,
title,
summary,
author,
feed_id,
likes,
dislikes,
love,
anger,
tags
FROM
feed_items fi
LEFT JOIN
feed_item_tags t
ON fi.feed_item_id = t.feed_item_id
LEFT JOIN
feed_item_love_anger_aggregate bba
ON fi.feed_item_id = bba.feed_item_id
LEFT JOIN
feed_item_likes_dislikes_aggregate lda
ON fi.feed_item_id = lda.feed_item_id
WHERE
(
pubdate,
fi.feed_item_id
)
< ('2020-06-19 19:50:00+05:30', 'bc5c8dfe-13a9-d97a-a328-0e5b8990c500')
AND
(
title_vector @@ to_tsquery('scam')
OR summary_vector @@ to_tsquery('scam')
)
AND 'for' = ANY(tags)
AND likes > 0
ORDER BY
pubdate DESC,
feed_item_id DESC LIMIT 3;
解释第 N 页查询 尽管进行了过滤，但它仍在进行 2 次顺序扫描
Limit (cost=2.83..17.13 rows=3 width=233) (actual time=0.082..0.199 rows=3 loops=1)
-> Nested Loop Left Join (cost=2.83..121.97 rows=25 width=233) (actual time=0.081..0.198 rows=3 loops=1)
-> Nested Loop (cost=2.69..113.67 rows=25 width=217) (actual time=0.073..0.185 rows=3 loops=1)
Join Filter: (t.feed_item_id = fi.feed_item_id)
Rows Removed by Join Filter: 183
-> Index Scan using idx_feed_items_pubdate_feed_item_id_desc on feed_items fi (cost=0.14..67.45 rows=74 width=62) (actual time=0.014..0.034 rows=6 loops=1)
Index Cond: (ROW(pubdate, feed_item_id) < ROW('2020-06-19 19:50:00+05:30'::timestamp with time zone, 'bc5c8dfe-13a9-d97a-a328-0e5b8990c500'::uuid))
Filter: ((title_vector @@ to_tsquery('scam'::text)) OR (summary_vector @@ to_tsquery('scam'::text)))
Rows Removed by Filter: 2
-> Materialize (cost=2.55..8.56 rows=34 width=187) (actual time=0.009..0.022 rows=31 loops=6)
-> Hash Join (cost=2.55..8.39 rows=34 width=187) (actual time=0.050..0.098 rows=36 loops=1)
Hash Cond: (t.feed_item_id = lda.feed_item_id)
-> Seq Scan on feed_item_tags t (cost=0.00..5.25 rows=67 width=155) (actual time=0.009..0.044 rows=67 loops=1)
Filter: ('for'::text = ANY ((tags)::text[]))
Rows Removed by Filter: 33
-> Hash (cost=1.93..1.93 rows=50 width=32) (actual time=0.028..0.029 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on feed_item_likes_dislikes_aggregate lda (cost=0.00..1.93 rows=50 width=32) (actual time=0.005..0.014 rows=50 loops=1)
Filter: (likes > 0)
Rows Removed by Filter: 24
-> Index Scan using idx_feed_item_love_anger_aggregate on feed_item_love_anger_aggregate bba (cost=0.14..0.32 rows=1 width=32) (actual time=0.003..0.003 rows=1 loops=3)
Index Cond: (feed_item_id = fi.feed_item_id)
Planning Time: 0.596 ms
Execution Time: 0.236 ms
(24 rows)
我已经用所需的 table 和索引设置了 fiddle，有人可以告诉我如何修复查询以最好或减少使用索引扫描顺序扫描的数量为 1?
结构'for' = ANY(tags)不能使用GIN索引。为了能够使用它，您需要将其重新表述为 '{for}' <@ tags.
然而，它会选择不使用索引，因为 table 太小，条件太无选择性。如果你想强制使用索引，为了证明它能够这样做，你可以先 set enable_seqscan=off.
除了 GIN 索引，您目前在标签 table 上没有其他索引。在你的 fiddle 中，如果我 create index on feed_item_tags (feed_item_id) 并执行 ANALYZE，那么两个序列扫描都会消失。这样做可能比重新制定更好，以便它可以使用 GIN 索引，就像我的其他答案一样，因为这种方式可以更有效地利用 LIMIT 提前停止的前景。
但实际上，“feed_item_tags”table 的意义何在？如果您要使用 child table 来列出标签，通常每行会有一个 tag/parent_id 组合。如果您想要一个标签数组而不是一列标签，为什么不直接将数组粘贴到 parent table 中呢？有时 table 与两个 table 之间的 1:1 关系是有原因的，但并不常见。