MySQL - 最近加入时出现性能问题
MySQL - Performance issue while joining most recent
我有两个表，markets（27 条记录）和 histories（约 175 万条记录，每个市场约 67K）。
我需要获取每个市场的最新 histories 记录。
我尝试过的解决方案有效但速度非常慢。
表 DDL
CREATE TABLE `markets` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`base_asset_id` int unsigned NOT NULL,
`quote_asset_id` int unsigned NOT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `markets_base_asset_id_quote_asset_id_unique` (`base_asset_id`,`quote_asset_id`),
KEY `markets_base_asset_id_index` (`base_asset_id`),
KEY `markets_quote_asset_id_index` (`quote_asset_id`),
CONSTRAINT `markets_base_asset_id_foreign` FOREIGN KEY (`base_asset_id`) REFERENCES `assets` (`id`),
CONSTRAINT `markets_quote_asset_id_foreign` FOREIGN KEY (`quote_asset_id`) REFERENCES `assets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
CREATE TABLE `histories` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`market_id` int unsigned NOT NULL,
`timeframe` enum('1_m','5_m','15_m','30_m','1_H','4_H','6_H','12_H','1_D','1_W','1_M') CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
`time` int unsigned NOT NULL,
`is_source` tinyint(1) NOT NULL DEFAULT '0',
`is_final` tinyint(1) NOT NULL DEFAULT '0',
`open` decimal(36,18) NOT NULL,
`high` decimal(36,18) NOT NULL,
`low` decimal(36,18) NOT NULL,
`close` decimal(36,18) DEFAULT NULL,
`volume` decimal(36,18) NOT NULL,
`ohlc_avg` decimal(36,18) DEFAULT NULL,
`hl_avg` decimal(36,18) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `histories_market_id_timeframe_time_unique` (`market_id`,`timeframe`,`time`),
KEY `histories_market_id_index` (`market_id`),
KEY `histories_timeframe_index` (`timeframe`),
KEY `histories_time_index` (`time`) USING BTREE,
CONSTRAINT `histories_market_id_foreign` FOREIGN KEY (`market_id`) REFERENCES `markets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2334503 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
我试过的
1 - 不相关的子查询
我从这个解决方案开始，因为我以前用过它，它需要 ~7.5s:
SELECT
m.*,
h.time,
h.close
FROM
markets m
LEFT JOIN (
SELECT
market_id,
MAX(`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
) latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
and h.`time` = latest_history.time;
EXPLAIN 结果：
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index,histories_time_index | histories_time_index | 4 | latest_history.time | 26 | 100.00 | Using where |
| 2 | DERIVED | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
2 - 使用 WITH
我尝试使用 WITH 运行 子查询，但没有任何改进，仍然 ~7.5s:
WITH latest_history AS (
SELECT
market_id,
MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
)
SELECT
m.*,
h.time,
h.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
AND h.`time` = latest_history.time;
EXPLAIN 结果（与上一个相同）
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index,histories_time_index | histories_time_index | 4 | latest_history.time | 26 | 100.00 | Using where |
| 2 | DERIVED | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
3 - 使用 WITH 和 window 函数
然后我从 5.7 升级到 8.0.22 以尝试其他建议的方法，这需要更多时间：~11s
WITH latest_history AS (
SELECT
h.*,
ROW_NUMBER() OVER (PARTITION BY market_id ORDER BY id DESC) AS rn
FROM
histories AS h
where
h.is_final = 1
)
SELECT
m.*,
latest_history.time,
latest_history.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
AND latest_history.rn = 1;
EXPLAIN 结果：
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 12 | db_name.m.id,const | 1744 | 100.00 | NULL |
| 2 | DERIVED | h | NULL | ALL | NULL | NULL | NULL | NULL | 1744647 | 10.00 | Using where; Using filesort |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
附加信息
然后我看到解决方案 1 和 2[=80 中需要单独的子查询（MAX 和 GROUP BY） =],耗时~7.5s!
所以我认为 histories structure/indexes 应该存在根本性的错误，而不是我加入 markets 的方式。
明确地说，这就是我指的那个：
SELECT
market_id,
MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id;
EXPLAIN 结果：
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| 1 | SIMPLE | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
time 是一个表示 Unix 时间戳的整数，可以使用 id 但它不会提高性能。
当然，我尝试查看其他 questions/posts 关于此组的性能问题，但我仍然无法修复它。
一种替代方法是使用子查询进行过滤：
select m.*, h.time, h.close
from markets m
left join histories h
on h.market_id = m.id
and h.time = (
select max(h1.time) from histories h1 where h1.market_id = m.id and h1.is_final = 1
)
为了性能，您需要在 histories(market_id, is_final, time desc) 上建立索引。
由于您只需要 histories table 中的两列，您可能还需要考虑使用两个子查询：
select m.*,
(select h.time from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as time,
(select h.close from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as close
from markets m
相同的索引将有助于查询 - 我们甚至可以在索引末尾添加 close，因此：histories(market_id, is_final, time desc, close).
最后：在 MySQL 的最新版本中，您可以尝试横向连接：
select m.*, h.*
from markets m
left join lateral (
select h.time, h.close
from history h
where h.market_id = m.id and h.is_final = 1
order by h.time desc limit 1
) h on true
我有两个表，markets（27 条记录）和 histories（约 175 万条记录，每个市场约 67K）。
我需要获取每个市场的最新 histories 记录。
我尝试过的解决方案有效但速度非常慢。
表 DDL
CREATE TABLE `markets` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`base_asset_id` int unsigned NOT NULL,
`quote_asset_id` int unsigned NOT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `markets_base_asset_id_quote_asset_id_unique` (`base_asset_id`,`quote_asset_id`),
KEY `markets_base_asset_id_index` (`base_asset_id`),
KEY `markets_quote_asset_id_index` (`quote_asset_id`),
CONSTRAINT `markets_base_asset_id_foreign` FOREIGN KEY (`base_asset_id`) REFERENCES `assets` (`id`),
CONSTRAINT `markets_quote_asset_id_foreign` FOREIGN KEY (`quote_asset_id`) REFERENCES `assets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
CREATE TABLE `histories` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`market_id` int unsigned NOT NULL,
`timeframe` enum('1_m','5_m','15_m','30_m','1_H','4_H','6_H','12_H','1_D','1_W','1_M') CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
`time` int unsigned NOT NULL,
`is_source` tinyint(1) NOT NULL DEFAULT '0',
`is_final` tinyint(1) NOT NULL DEFAULT '0',
`open` decimal(36,18) NOT NULL,
`high` decimal(36,18) NOT NULL,
`low` decimal(36,18) NOT NULL,
`close` decimal(36,18) DEFAULT NULL,
`volume` decimal(36,18) NOT NULL,
`ohlc_avg` decimal(36,18) DEFAULT NULL,
`hl_avg` decimal(36,18) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `histories_market_id_timeframe_time_unique` (`market_id`,`timeframe`,`time`),
KEY `histories_market_id_index` (`market_id`),
KEY `histories_timeframe_index` (`timeframe`),
KEY `histories_time_index` (`time`) USING BTREE,
CONSTRAINT `histories_market_id_foreign` FOREIGN KEY (`market_id`) REFERENCES `markets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2334503 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
我试过的
1 - 不相关的子查询
我从这个解决方案开始，因为我以前用过它，它需要 ~7.5s:
SELECT
m.*,
h.time,
h.close
FROM
markets m
LEFT JOIN (
SELECT
market_id,
MAX(`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
) latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
and h.`time` = latest_history.time;
EXPLAIN 结果：
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index,histories_time_index | histories_time_index | 4 | latest_history.time | 26 | 100.00 | Using where |
| 2 | DERIVED | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
2 - 使用 WITH
我尝试使用 WITH 运行 子查询，但没有任何改进，仍然 ~7.5s:
WITH latest_history AS (
SELECT
market_id,
MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
)
SELECT
m.*,
h.time,
h.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
AND h.`time` = latest_history.time;
EXPLAIN 结果（与上一个相同）
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index,histories_time_index | histories_time_index | 4 | latest_history.time | 26 | 100.00 | Using where |
| 2 | DERIVED | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
3 - 使用 WITH 和 window 函数
然后我从 5.7 升级到 8.0.22 以尝试其他建议的方法，这需要更多时间：~11s
WITH latest_history AS (
SELECT
h.*,
ROW_NUMBER() OVER (PARTITION BY market_id ORDER BY id DESC) AS rn
FROM
histories AS h
where
h.is_final = 1
)
SELECT
m.*,
latest_history.time,
latest_history.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
AND latest_history.rn = 1;
EXPLAIN 结果：
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 12 | db_name.m.id,const | 1744 | 100.00 | NULL |
| 2 | DERIVED | h | NULL | ALL | NULL | NULL | NULL | NULL | 1744647 | 10.00 | Using where; Using filesort |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
附加信息
然后我看到解决方案 1 和 2[=80 中需要单独的子查询（MAX 和 GROUP BY） =],耗时~7.5s!
所以我认为 histories structure/indexes 应该存在根本性的错误，而不是我加入 markets 的方式。
明确地说，这就是我指的那个：
SELECT
market_id,
MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id;
EXPLAIN 结果：
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| 1 | SIMPLE | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
time 是一个表示 Unix 时间戳的整数，可以使用 id 但它不会提高性能。
当然，我尝试查看其他 questions/posts 关于此组的性能问题，但我仍然无法修复它。
一种替代方法是使用子查询进行过滤：
select m.*, h.time, h.close
from markets m
left join histories h
on h.market_id = m.id
and h.time = (
select max(h1.time) from histories h1 where h1.market_id = m.id and h1.is_final = 1
)
为了性能，您需要在 histories(market_id, is_final, time desc) 上建立索引。
由于您只需要 histories table 中的两列，您可能还需要考虑使用两个子查询：
select m.*,
(select h.time from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as time,
(select h.close from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as close
from markets m
相同的索引将有助于查询 - 我们甚至可以在索引末尾添加 close，因此：histories(market_id, is_final, time desc, close).
最后：在 MySQL 的最新版本中，您可以尝试横向连接：
select m.*, h.*
from markets m
left join lateral (
select h.time, h.close
from history h
where h.market_id = m.id and h.is_final = 1
order by h.time desc limit 1
) h on true