Postgres SQL 大查询速度慢 table (AWS RDS)
Postgres SQL query slow with large table (AWS RDS)
目前table的最小行数为3000万,而且还在增长,每当尝试做SELECT查询时,都需要非常长的时间。在提高数据库性能之前需要优化什么查询?
POSTGRES 12 on AWS RDS db.t3.small, with 20GB storage
**Message Table**
id (bigint) -> pk
meta (jsonb)
snapshot_ts (integer) -> epoch timestamp
value (character varying 100)
type (character varying 50)
created (timestamp with timezone)
last_modified (timestamp with timezone)
attribute_id (bigint) -> Foreign Key
company_id (bigint) -> Foreign Key
project_id (bigint) -> Foreign Key
device_id (bigint) -> Foreign Key
EXPLAIN (analyze,buffers) SELECT COUNT(*) FROM public.message
WHERE company_id=446 AND project_id=52 AND snapshot_ts>=1637568000.0 AND snapshot_ts<=1637654399.0 AND attribute_id=458
->Aggregate (cost=399804.26..399804.27 rows=1 width=8) (actual time=65150.696..65150.697 rows=1 loops=1)
Buffers: shared hit=170 read=115437 dirtied=167
I/O Timings: read=64396.424
-> Index Scan using message_attribute_id_6578b282 on message (cost=0.56..399803.23 rows=411 width=0) (actual time=57752.297..65147.391 rows=8656 loops=1)
Index Cond: (attribute_id = 458)
Filter: ((company_id = 446) AND (project_id = 52) AND ((snapshot_ts)::numeric >= 1637568000.0) AND ((snapshot_ts)::numeric <= 1637654399.0))
Rows Removed by Filter: 106703
Buffers: shared hit=170 read=115437 dirtied=167
I/O Timings: read=64396.424
Planning Time: 0.779 ms
Execution Time: 65150.730 ms
**Indexes**
indexname | indexdef
message_attribute_id_6578b282 | CREATE INDEX message_attribute_id_6578b282 ON public.message USING btree (attribute_id)
message_company_id_cef5ed5f | CREATE INDEX message_company_id_cef5ed5f ON public.message USING btree (company_id)
message_device_id_b4da2571 | CREATE INDEX message_device_id_b4da2571 ON public.message USING btree (device_id)
message_pkey | CREATE UNIQUE INDEX message_pkey ON public.message USING btree (id)
message_project_id_7ba6787d | CREATE INDEX message_project_id_7ba6787d ON public.message USING btree (project_id)
考虑具体查询:
SELECT COUNT(*)
FROM public.message
WHERE company_id=446
AND project_id=52
AND snapshot_ts>=1637568000.0 AND snapshot_ts<=1637654399.0
AND attribute_id=458
以下索引具有极大提高性能的潜力:
create index ix1 on public.message (
company_id, project_id, attribute_id, snapshot_ts
);
但是,请记住,在 3000 万行上创建索引 table 可能需要一些时间。
目前table的最小行数为3000万,而且还在增长,每当尝试做SELECT查询时,都需要非常长的时间。在提高数据库性能之前需要优化什么查询?
POSTGRES 12 on AWS RDS db.t3.small, with 20GB storage
**Message Table**
id (bigint) -> pk
meta (jsonb)
snapshot_ts (integer) -> epoch timestamp
value (character varying 100)
type (character varying 50)
created (timestamp with timezone)
last_modified (timestamp with timezone)
attribute_id (bigint) -> Foreign Key
company_id (bigint) -> Foreign Key
project_id (bigint) -> Foreign Key
device_id (bigint) -> Foreign Key
EXPLAIN (analyze,buffers) SELECT COUNT(*) FROM public.message
WHERE company_id=446 AND project_id=52 AND snapshot_ts>=1637568000.0 AND snapshot_ts<=1637654399.0 AND attribute_id=458
->Aggregate (cost=399804.26..399804.27 rows=1 width=8) (actual time=65150.696..65150.697 rows=1 loops=1)
Buffers: shared hit=170 read=115437 dirtied=167
I/O Timings: read=64396.424
-> Index Scan using message_attribute_id_6578b282 on message (cost=0.56..399803.23 rows=411 width=0) (actual time=57752.297..65147.391 rows=8656 loops=1)
Index Cond: (attribute_id = 458)
Filter: ((company_id = 446) AND (project_id = 52) AND ((snapshot_ts)::numeric >= 1637568000.0) AND ((snapshot_ts)::numeric <= 1637654399.0))
Rows Removed by Filter: 106703
Buffers: shared hit=170 read=115437 dirtied=167
I/O Timings: read=64396.424
Planning Time: 0.779 ms
Execution Time: 65150.730 ms
**Indexes**
indexname | indexdef
message_attribute_id_6578b282 | CREATE INDEX message_attribute_id_6578b282 ON public.message USING btree (attribute_id)
message_company_id_cef5ed5f | CREATE INDEX message_company_id_cef5ed5f ON public.message USING btree (company_id)
message_device_id_b4da2571 | CREATE INDEX message_device_id_b4da2571 ON public.message USING btree (device_id)
message_pkey | CREATE UNIQUE INDEX message_pkey ON public.message USING btree (id)
message_project_id_7ba6787d | CREATE INDEX message_project_id_7ba6787d ON public.message USING btree (project_id)
考虑具体查询:
SELECT COUNT(*)
FROM public.message
WHERE company_id=446
AND project_id=52
AND snapshot_ts>=1637568000.0 AND snapshot_ts<=1637654399.0
AND attribute_id=458
以下索引具有极大提高性能的潜力:
create index ix1 on public.message (
company_id, project_id, attribute_id, snapshot_ts
);
但是,请记住,在 3000 万行上创建索引 table 可能需要一些时间。