在非不同索引上使用递归 cte 计算不同行

Question

给定以下架构：

CREATE TABLE identifiers (
  id TEXT PRIMARY KEY
);

CREATE TABLE days (
  day DATE PRIMARY KEY
);

CREATE TABLE data (
  id TEXT REFERENCES identifiers
  , day DATE REFERENCES days
  , values NUMERIC[] 
); 
CREATE INDEX ON data (id, day);

计算两个时间戳之间所有不同天数的最佳方法是什么？我尝试了以下两种方法：

EXPLAIN ANALYZE
SELECT COUNT(DISTINCT day) 
FROM data 
WHERE day BETWEEN '2010-01-01' AND '2011-01-01';
                                                                        QUERY PLAN                                                                        
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=200331.32..200331.33 rows=1 width=4) (actual time=1647.574..1647.575 rows=1 loops=1)
   ->  Index Only Scan using data_day_sid_idx on data  (cost=0.56..196942.12 rows=1355678 width=4) (actual time=0.348..1180.566 rows=1362532 loops=1)
         Index Cond: ((day >= '2010-01-01'::date) AND (day <= '2011-01-01'::date))
         Heap Fetches: 0
 Total runtime: 1647.865 ms
(5 rows)

EXPLAIN ANALYZE
SELECT COUNT(DISTINCT day) 
FROM days
WHERE day BETWEEN '2010-01-01' AND '2011-01-01';
                                                           QUERY PLAN                                                           
--------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=18.95..18.96 rows=1 width=4) (actual time=0.481..0.481 rows=1 loops=1)
   ->  Index Only Scan using days_pkey on days  (cost=0.28..18.32 rows=252 width=4) (actual time=0.093..0.275 rows=252 loops=1)
         Index Cond: ((day >= '2010-01-01'::date) AND (day <= '2011-01-01'::date))
         Heap Fetches: 252
 Total runtime: 0.582 ms
(5 rows)

COUNT(DISTINCT day)对days运行很好，但它需要我保持辅助table（days）以保持合理的性能.一般来说，我想测试递归 cte 是否能让我实现类似的性能 而无需 维护辅助 table。我的查询看起来像这样，但运行还没有：

EXPLAIN ANALYZE
WITH RECURSIVE cte AS (
   (SELECT day FROM data ORDER BY 1 LIMIT 1)
   UNION ALL
   (  -- parentheses required
   SELECT d.day
   FROM   cte  c
   JOIN   data d ON d.day > c.day
   ORDER  BY 1 LIMIT 1
   )
)
SELECT day 
FROM cte
WHERE day BETWEEN '2010-01-01' AND '2011-01-01';

更新

感谢大家的想法。看起来维护不同日期的基于触发器的 table 是最好的方法，无论是存储还是性能方面。感谢@Erwin 的更新，递归 CTE 又回到了运行ning。很有用。

WITH RECURSIVE cte AS (
   (  -- parentheses required because of LIMIT
   SELECT day
   FROM   data
   WHERE  day >= '2010-01-01'::date  -- exclude irrelevant rows early
   ORDER  BY 1
   LIMIT  1
   )

   UNION ALL
   SELECT (SELECT day FROM data
           WHERE  day > c.day
           AND    day < '2011-01-01'::date  -- see comments below
           ORDER  BY 1
           LIMIT  1)
   FROM   cte c
   WHERE  day IS NOT NULL  -- necessary because corr. subq. always returns row
   )
SELECT count(*) AS ct
FROM   cte
WHERE  day IS NOT NULL;

                                                                             QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=53.35..53.36 rows=1 width=0) (actual time=18.217..18.217 rows=1 loops=1)
   CTE cte
     ->  Recursive Union  (cost=0.43..51.08 rows=101 width=4) (actual time=0.194..17.594 rows=253 loops=1)
           ->  Limit  (cost=0.43..0.46 rows=1 width=4) (actual time=0.191..0.192 rows=1 loops=1)
                 ->  Index Only Scan using data_day_idx on data data_1  (cost=0.43..235042.00 rows=8255861 width=4) (actual time=0.189..0.189 rows=1 loops=1)
                       Index Cond: (day >= '2010-01-01'::date)
                       Heap Fetches: 0
           ->  WorkTable Scan on cte c  (cost=0.00..4.86 rows=10 width=4) (actual time=0.066..0.066 rows=1 loops=253)
                 Filter: (day IS NOT NULL)
                 Rows Removed by Filter: 0
                 SubPlan 1
                   ->  Limit  (cost=0.43..0.47 rows=1 width=4) (actual time=0.062..0.063 rows=1 loops=252)
                         ->  Index Only Scan using data_day_idx on data  (cost=0.43..1625.59 rows=52458 width=4) (actual time=0.060..0.060 rows=1 loops=252)
                               Index Cond: ((day > c.day) AND (day < '2011-01-01'::date))
                               Heap Fetches: 0
   ->  CTE Scan on cte  (cost=0.00..2.02 rows=100 width=0) (actual time=0.199..18.066 rows=252 loops=1)
         Filter: (day IS NOT NULL)
         Rows Removed by Filter: 1
 Total runtime: 19.355 ms
(19 rows)

并且还讨论了 EXISTS 查询

EXPLAIN ANALYZE
SELECT count(*) AS ct
FROM   generate_series('2010-01-01'::date, '2010-12-31'::date, '1d'::interval) d(day)
WHERE  EXISTS (SELECT 1 FROM data WHERE day = d.day::date);

                                                                  QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=674.32..674.33 rows=1 width=0) (actual time=95.049..95.049 rows=1 loops=1)
   ->  Nested Loop Semi Join  (cost=0.45..673.07 rows=500 width=0) (actual time=12.438..94.749 rows=252 loops=1)
         ->  Function Scan on generate_series d  (cost=0.01..10.01 rows=1000 width=8) (actual time=9.248..9.669 rows=365 loops=1)
         ->  Index Only Scan using data_day_idx on data  (cost=0.44..189.62 rows=6023 width=4) (actual time=0.227..0.227 rows=1 loops=365)
               Index Cond: (day = (d.day)::date)
               Heap Fetches: 0
 Total runtime: 95.620 ms
(7 rows)

Answer 1

尝试在 data(day) 上创建索引，然后运行第一个查询：

SELECT COUNT(DISTINCT day) 
FROM data 
WHERE day BETWEEN '2010-01-01' AND '2011-01-01';

您可能会发现性能足以满足您的目的。

Answer 2

几个注意事项：

table `day`

上的简单查询

SELECT COUNT(<strike>DISTINCT</strike> day) 
FROM   days
WHERE  day BETWEEN '2010-01-01' AND '2011-01-01';

虽然day被定义为PK，DISTINCT只是昂贵的噪音。

具有相关suquery的递归CTE

如果 没有 day table 具有唯一条目，则这是备选方案。如果每天有多行到多行，则该技术是值得的，因此相当于松散索引扫描实际上比基于 table:

的简单 DISTINCT 更快

WITH RECURSIVE cte AS (
   (  -- parentheses required because of LIMIT
   SELECT day
   FROM   data
   <b>WHERE  day >= '2010-01-01'</b>  -- exclude irrelevant rows early
   ORDER  BY 1
   LIMIT  1
   )
  
   UNION ALL
   SELECT (SELECT day FROM data
           WHERE  day > c.day
           <b>AND    day < '2011-01-01'</b>  -- see below
           ORDER  BY 1
           LIMIT  1)
   FROM   cte c
   WHERE  day IS NOT NULL  -- necessary because corr. subq. always returns row
   )
SELECT count(*) AS ct
FROM   cte
WHERE  day IS NOT NULL;

索引

只有与 data 上的匹配索引结合才有意义：

CREATE INDEX data_day_idx ON data (day);

day 必须是前导列。您在 (id, day) 问题中的索引也可以使用，但效率要低得多：

备注

尽早排除不相关的行要便宜得多。我将你的谓词整合到查询中。

详细解释：

Optimize GROUP BY query to retrieve latest row per user

手头的案例更简单 - 实际上是最简单的。

您原来的时间范围是 day BETWEEN '2010-01-01' AND '2011-01-01'。但是 BETWEEN .. AND .. 包括上限和下限，因此您将获得 2010 年的全部加上 2011-01-01。您可能想要排除上限。使用 d.day < '2011-01-01'（而不是 <=）。参见：

How to add a day/night indicator to a timestamp column?

`EXISTS` 对于这种特殊情况

由于您正在测试一系列可枚举的天数（与具有无限数量可能值的范围相反），您可以使用 EXISTS 半连接测试此替代方案：

SELECT count(*) AS ct
FROM   generate_series(timestamp '2010-01-01'
                     , timestamp '2010-12-31'
                     , interval  '1 day') AS d(day)
WHERE  EXISTS (SELECT FROM data WHERE day = d.day::date);

为什么这种形式的 generate_series() 是最佳的？

Generating time series between two dates in PostgreSQL

同样的简单索引又是必不可少的。

db<>fiddle here 用大测试证明两者 table.
_旧sqlfiddle

Answer 3

我不太确定为什么 data(day) 上的索引较慢，这似乎是最简单的选择。但如果这太慢了，您可以尝试创建一个具体化的生活视图。基本上只是：

create materialized view days as
select day 
from data 
group by day;

我不相信 postgres 会自动更新物化视图，但至少您需要做的所有维护工作就是定期刷新它。或者也许创建一个刷新视图的数据触发器。当然请记住，刷新此视图可能需要一些时间，具体取决于数据的大小 table，如果可以的话，您可能只想每小时或每晚刷新一次。

或者，如果此 table 获得大量更新并且您需要不同的天数始终保持一致，您可以考虑回到原来的单独天数 table，但减少通过在数据 table 上创建触发器来更新它来减少维护开销。

在非不同索引上使用递归 cte 计算不同行

Counting distinct rows using recursive cte over non-distinct index

sql

postgresql

count

postgresql-9.3

unique-index

table `day`

具有相关suquery的递归CTE

索引

备注

`EXISTS` 对于这种特殊情况

在非不同索引上使用递归 cte 计算不同行

Counting distinct rows using recursive cte over non-distinct index

sql

postgresql

count

postgresql-9.3

unique-index

table day

具有相关suquery的递归CTE

索引

备注

EXISTS 对于这种特殊情况

table `day`

`EXISTS` 对于这种特殊情况