需要 POSTGRES 调优建议
Need suggestion for POSTGRES Tuning
我需要 运行 在我的数据库中频繁执行一些复杂的查询，我的一个查询执行大约需要 57 秒，我正在尝试一种调整我的 Postgres 数据库的方法，所以，我可以将执行时间缩短一半或更多。
我在 table 上创建了索引并调整了一些参数，但对我没有帮助。
我使用的是 Postgres 9.6 版本，下面是我用来调整数据库的参数。
- max_connections = 1500
- shared_buffers=8GB
- effective_cache_size = 8GB
- maintenance_work_mem = 2GB
- checkpoint_completion_target = 0.7
- wal_buffers = 16MB
- default_statistics_target = 100
- random_page_cost = 1.1
- effective_io_concurrency = 200
- work_mem = 910kB
- min_wal_size = 1GB
- max_wal_size = 4GB
- max_worker_processes = 48
- max_parallel_workers_per_gather = 24
查询如下
Explain Analyze SELECT a. SD_ID, a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID,
TO_CHAR(MIN(a.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'), SUM(CASE WHEN a.STATUS < 30 THEN a.THEO_SECONDS ELSE 0 END ), MAX(a.TRAIL_SECONDS), x. SD_ID, x.SIP_PART,
x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID, TO_CHAR(MIN(x.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'),
SUM(x.THEO_SECONDS), SUM(CASE WHEN x.STATUS >= 30 THEN x.THEO_SECONDS ELSE 0 END ), MAX(x.TRAIL_SECONDS),
MIN(a.PSEQ - x.PSEQ), CASE WHEN x. SD_ID <> a. SD_ID THEN MIN(x.BSEQ - a.BSEQ) ELSE 0 END, MIN(x.PSEQ), '', 0, 0, COUNT(1),
SUM(CASE WHEN x.STATUS >= 30 THEN 1 ELSE 0 END ) , CASE WHEN MIN(x.BSEQ) > 0 AND MIN(a.BSEQ) > 0 AND MIN(x.BSEQ) <> MIN(a.BSEQ) AND MIN(x.POS) = MIN(a.POS) THEN MIN(x.STEP - a.STEP) ELSE 0 END
FROM MYTESTABLE a, MYTESTABLE x
WHERE a.SIPD_DATE >= TO_TIMESTAMP('2020/07/20 00:00:00','YYYY/MM/DD HH24:MI:SS')
AND a.SIPD_DATE <= TO_TIMESTAMP('2020/07/20 23:59:59','YYYY/MM/DD HH24:MI:SS') AND a.HISTORY = 0 AND a.PSEQ > 0 AND x.HISTORY = 0 AND x.PSEQ > 0
AND a.ORDER_NO = x.ORDER_NO AND a.ITEM_NO = x.ITEM_NO AND a.QTY = x.QTY AND ( (a. SD_ID > 0 AND x. SD_ID > 0 AND a. SD_ID <> x. SD_ID
AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
OR (a. SD_ID = x. SD_ID AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
OR (a. SD_ID = x. SD_ID AND a.BSEQ = x.BSEQ AND ((a.PRODUCT_TYPE IS NULL AND x.PRODUCT_TYPE IS NULL)
OR (a.PRODUCT_TYPE IS NOT NULL AND a.PRODUCT_TYPE = x.PRODUCT_TYPE AND a.P_ID = x.P_ID))
AND (a.plan_notes <> a.plan_notes OR a.STEP_NO <> x.STEP_NO) AND a.PSEQ <> x.PSEQ)
OR (a. SD_ID = x. SD_ID AND a.BSEQ > 1 AND x.BSEQ > 1 AND a.BSEQ <> x.BSEQ AND a.PRODUCT_TYPE = 1 AND x.PRODUCT_TYPE = 1
AND a.P_ID <> x.P_ID AND a.POS = x.POS AND ((a.STEP = 0 AND x.STEP > 0) OR (x.STEP = 0 AND a.STEP > 0))))
GROUP BY a. SD_ID, a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID, DATE_TRUNC('day', a.SIPD_DATE) ,
x. SD_ID, x.SIP_PART, x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID, DATE_TRUNC('day', x.SIPD_DATE) ORDER BY a. SD_ID,
a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID, DATE_TRUNC('day', a.SIPD_DATE) , x. SD_ID, x.SIP_PART, MIN(x.PSEQ);
下面是查询计划
QUERY PLAN
Sort (cost=5426751.71..5436426.08 rows=3869747 width=482) (actual time=57121.003..57139.244 rows=105225 loops=1)
Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, (min(x.PSEQ))
Sort Method: external sort Disk: 19848kB
-> GroupAggregate (cost=1982498.13..2524262.71 rows=3869747 width=482) (actual time=34888.529..56516.255 rows=105225 loops=1)
Group Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID, (dat
e_trunc('day'::text, x.SIPd_date))
-> Sort (cost=1982498.13..1992172.50 rows=3869747 width=146) (actual time=34888.391..46834.281 rows=3195855 loops=1)
Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID,
(date_trunc('day'::text, x.SIPd_date))
Sort Method: external merge Disk: 519008kB
-> Gather (cost=1000.56..877326.14 rows=3869747 width=146) (actual time=10.146..1648.706 rows=3195855 loops=1)
Workers Planned: 6
Workers Launched: 6
-> Nested Loop (cost=0.56..473227.49 rows=644958 width=146) (actual time=10.876..5330.931 rows=456551 loops=7)
-> Parallel Seq Scan on MYTESTABLE a (cost=0.00..278565.05 rows=167365 width=82) (actual time=10.771..975.060 rows=79290 loops=7)
Filter: ((PSEQ > '0'::numeric) AND (history = '0'::numeric) AND (SIPd_date >= to_timestamp('2020/07/20 00:00:00'::text, 'YYYY/MM/DD HH24:MI:SS'::text)) AND (SIPd_date <= to_timestamp('2020/0
7/20 23:59:59'::text, 'YYYY/MM/DD HH24:MI:SS'::text)))
Rows Removed by Filter: 386673
-> Index Scan using idx_a1 on MYTESTABLE x (cost=0.56..1.13 rows=1 width=82) (actual time=0.020..0.049 rows=6 loops=555028)
Index Cond: ((order_no = a.order_no) AND (item_no = a.item_no) AND (QTY = a.QTY) AND (history = '0'::numeric) AND (PSEQ > '0'::numeric))
Filter: (((a. SD_ID > '0'::numeric) AND ( SD_ID > '0'::numeric) AND (a. SD_ID <> SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::nume
ric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID = SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::numeric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID = SIP_
id) AND (a.BSEQ = BSEQ) AND (((a.PRODUCT_TYPE IS NULL) AND (PRODUCT_TYPE IS NULL)) OR ((a.PRODUCT_TYPE IS NOT NULL) AND (a.PRODUCT_TYPE = PRODUCT_TYPE) AND (a.P_ID = P_ID))) AND ((a.plan_notes <> a.plan_notes) OR (a.step_no <> step_n
o)) AND (a.PSEQ <> PSEQ)) OR ((a. SD_ID = SD_ID) AND (a.BSEQ > '1'::numeric) AND (BSEQ > '1'::numeric) AND (a.BSEQ <> BSEQ) AND (a.PRODUCT_TYPE = '1'::numeric) AND (PRODUCT_TYPE = '1'::numeric) AND (a.mat_i
d <> P_ID) AND (a.POS = POS) AND (((a.STEP = '0'::numeric) AND (STEP > '0'::numeric)) OR ((STEP = '0'::numeric) AND (a.STEP > '0'::numeric)))))
Rows Removed by Filter: 3
Planning time: 9.241 ms
Execution time: 57280.673 ms
更新到 postgres10 后修复，jjanes 的建议有效
我需要 运行 在我的数据库中频繁执行一些复杂的查询，我的一个查询执行大约需要 57 秒，我正在尝试一种调整我的 Postgres 数据库的方法，所以，我可以将执行时间缩短一半或更多。
我在 table 上创建了索引并调整了一些参数，但对我没有帮助。
我使用的是 Postgres 9.6 版本，下面是我用来调整数据库的参数。
- max_connections = 1500
- shared_buffers=8GB
- effective_cache_size = 8GB
- maintenance_work_mem = 2GB
- checkpoint_completion_target = 0.7
- wal_buffers = 16MB
- default_statistics_target = 100
- random_page_cost = 1.1
- effective_io_concurrency = 200
- work_mem = 910kB
- min_wal_size = 1GB
- max_wal_size = 4GB
- max_worker_processes = 48
- max_parallel_workers_per_gather = 24
查询如下
Explain Analyze SELECT a. SD_ID, a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID,
TO_CHAR(MIN(a.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'), SUM(CASE WHEN a.STATUS < 30 THEN a.THEO_SECONDS ELSE 0 END ), MAX(a.TRAIL_SECONDS), x. SD_ID, x.SIP_PART,
x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID, TO_CHAR(MIN(x.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'),
SUM(x.THEO_SECONDS), SUM(CASE WHEN x.STATUS >= 30 THEN x.THEO_SECONDS ELSE 0 END ), MAX(x.TRAIL_SECONDS),
MIN(a.PSEQ - x.PSEQ), CASE WHEN x. SD_ID <> a. SD_ID THEN MIN(x.BSEQ - a.BSEQ) ELSE 0 END, MIN(x.PSEQ), '', 0, 0, COUNT(1),
SUM(CASE WHEN x.STATUS >= 30 THEN 1 ELSE 0 END ) , CASE WHEN MIN(x.BSEQ) > 0 AND MIN(a.BSEQ) > 0 AND MIN(x.BSEQ) <> MIN(a.BSEQ) AND MIN(x.POS) = MIN(a.POS) THEN MIN(x.STEP - a.STEP) ELSE 0 END
FROM MYTESTABLE a, MYTESTABLE x
WHERE a.SIPD_DATE >= TO_TIMESTAMP('2020/07/20 00:00:00','YYYY/MM/DD HH24:MI:SS')
AND a.SIPD_DATE <= TO_TIMESTAMP('2020/07/20 23:59:59','YYYY/MM/DD HH24:MI:SS') AND a.HISTORY = 0 AND a.PSEQ > 0 AND x.HISTORY = 0 AND x.PSEQ > 0
AND a.ORDER_NO = x.ORDER_NO AND a.ITEM_NO = x.ITEM_NO AND a.QTY = x.QTY AND ( (a. SD_ID > 0 AND x. SD_ID > 0 AND a. SD_ID <> x. SD_ID
AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
OR (a. SD_ID = x. SD_ID AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
OR (a. SD_ID = x. SD_ID AND a.BSEQ = x.BSEQ AND ((a.PRODUCT_TYPE IS NULL AND x.PRODUCT_TYPE IS NULL)
OR (a.PRODUCT_TYPE IS NOT NULL AND a.PRODUCT_TYPE = x.PRODUCT_TYPE AND a.P_ID = x.P_ID))
AND (a.plan_notes <> a.plan_notes OR a.STEP_NO <> x.STEP_NO) AND a.PSEQ <> x.PSEQ)
OR (a. SD_ID = x. SD_ID AND a.BSEQ > 1 AND x.BSEQ > 1 AND a.BSEQ <> x.BSEQ AND a.PRODUCT_TYPE = 1 AND x.PRODUCT_TYPE = 1
AND a.P_ID <> x.P_ID AND a.POS = x.POS AND ((a.STEP = 0 AND x.STEP > 0) OR (x.STEP = 0 AND a.STEP > 0))))
GROUP BY a. SD_ID, a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID, DATE_TRUNC('day', a.SIPD_DATE) ,
x. SD_ID, x.SIP_PART, x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID, DATE_TRUNC('day', x.SIPD_DATE) ORDER BY a. SD_ID,
a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID, DATE_TRUNC('day', a.SIPD_DATE) , x. SD_ID, x.SIP_PART, MIN(x.PSEQ);
下面是查询计划
QUERY PLAN
Sort (cost=5426751.71..5436426.08 rows=3869747 width=482) (actual time=57121.003..57139.244 rows=105225 loops=1)
Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, (min(x.PSEQ))
Sort Method: external sort Disk: 19848kB
-> GroupAggregate (cost=1982498.13..2524262.71 rows=3869747 width=482) (actual time=34888.529..56516.255 rows=105225 loops=1)
Group Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID, (dat
e_trunc('day'::text, x.SIPd_date))
-> Sort (cost=1982498.13..1992172.50 rows=3869747 width=146) (actual time=34888.391..46834.281 rows=3195855 loops=1)
Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID,
(date_trunc('day'::text, x.SIPd_date))
Sort Method: external merge Disk: 519008kB
-> Gather (cost=1000.56..877326.14 rows=3869747 width=146) (actual time=10.146..1648.706 rows=3195855 loops=1)
Workers Planned: 6
Workers Launched: 6
-> Nested Loop (cost=0.56..473227.49 rows=644958 width=146) (actual time=10.876..5330.931 rows=456551 loops=7)
-> Parallel Seq Scan on MYTESTABLE a (cost=0.00..278565.05 rows=167365 width=82) (actual time=10.771..975.060 rows=79290 loops=7)
Filter: ((PSEQ > '0'::numeric) AND (history = '0'::numeric) AND (SIPd_date >= to_timestamp('2020/07/20 00:00:00'::text, 'YYYY/MM/DD HH24:MI:SS'::text)) AND (SIPd_date <= to_timestamp('2020/0
7/20 23:59:59'::text, 'YYYY/MM/DD HH24:MI:SS'::text)))
Rows Removed by Filter: 386673
-> Index Scan using idx_a1 on MYTESTABLE x (cost=0.56..1.13 rows=1 width=82) (actual time=0.020..0.049 rows=6 loops=555028)
Index Cond: ((order_no = a.order_no) AND (item_no = a.item_no) AND (QTY = a.QTY) AND (history = '0'::numeric) AND (PSEQ > '0'::numeric))
Filter: (((a. SD_ID > '0'::numeric) AND ( SD_ID > '0'::numeric) AND (a. SD_ID <> SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::nume
ric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID = SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::numeric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID = SIP_
id) AND (a.BSEQ = BSEQ) AND (((a.PRODUCT_TYPE IS NULL) AND (PRODUCT_TYPE IS NULL)) OR ((a.PRODUCT_TYPE IS NOT NULL) AND (a.PRODUCT_TYPE = PRODUCT_TYPE) AND (a.P_ID = P_ID))) AND ((a.plan_notes <> a.plan_notes) OR (a.step_no <> step_n
o)) AND (a.PSEQ <> PSEQ)) OR ((a. SD_ID = SD_ID) AND (a.BSEQ > '1'::numeric) AND (BSEQ > '1'::numeric) AND (a.BSEQ <> BSEQ) AND (a.PRODUCT_TYPE = '1'::numeric) AND (PRODUCT_TYPE = '1'::numeric) AND (a.mat_i
d <> P_ID) AND (a.POS = POS) AND (((a.STEP = '0'::numeric) AND (STEP > '0'::numeric)) OR ((STEP = '0'::numeric) AND (a.STEP > '0'::numeric)))))
Rows Removed by Filter: 3
Planning time: 9.241 ms
Execution time: 57280.673 ms
更新到 postgres10 后修复，jjanes 的建议有效