在 MySQL 5.7 中优化查询
optimize a query in MySQL 5.7
这是我的系统：
Linux Ubuntu 18.04 LTS
MySQL 5.7
我有一个查询（如下）需要很长时间才能完成。
完成需要 9 秒。在等待网页完成加载时，对于用户来说，这是一段难以承受的时间。
此外，数据集很大而且还在增长。事件 table 有 250,000 行。我倾向于每天添加 1200 到 1800 行。
为了帮助优化，我想添加索引等，但我不知道如何（以及是否）可以使用我的 derived/joined select 查询来做到这一点。
这是我的查询。 （有谁知道如何限制输出的宽度，这样我们就不必向右滚动就能看到整行了？）
mysql> explain select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname
AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 263103 | 100.00 | Using where |
| 1 | PRIMARY | OUTSIDE | NULL | ref | camtimestamp,camname | camtimestamp | 6 | INSIDE.maxtimestamp | 1 | 99.73 | Using where |
| 2 | DERIVED | events | NULL | index | camname | camname | 257 | NULL | 263103 | 100.00 | NULL |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
3 rows in set, 1 warning (0.03 sec)
查询结果如下：
mysql> select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname
AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time | Now | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| 1 | 1 | wls1 | In | 2020-01-30 12:27:31 | 2020-01-30 12:29:53 | 00:03:42 |
| 1 | 1 | wls2 | Out | 2020-01-30 12:25:29 | 2020-01-30 12:29:53 | 00:07:04 |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (6.49 sec)
============================================= =============
感谢所有回答问题或发表评论的人。我采纳了 Uueerdo 关于复合索引的建议，得出了以下结果：
mysql> create index CamNameCamTime on events (camname,camtimestamp);
mysql> select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp
from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time | Now | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| 1 | 1 | wls1 | In | 2020-01-30 18:43:19 | 2020-01-30 18:44:33 | 00:01:54 |
| 1 | 1 | wls2 | Out | 2020-01-30 18:41:51 | 2020-01-30 18:44:33 | 00:04:42 |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (0.00 sec)
mysql> explain select OUTSIDE.ownerUID as Owner,
-> OUTSIDE.propUID as Property,
-> OUTSIDE.camname as 'Camera Name',
-> OUTSIDE.direction as Direction,
-> OUTSIDE.camtimestamp as 'Event Time',
-> convert_tz(now(),'UTC','US/Central') as Now,
-> sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
-> from events OUTSIDE,
-> (select camname,max(camtimestamp) as maxtimestamp
-> from events group by camname) as INSIDE
-> where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | Using where |
| 1 | PRIMARY | OUTSIDE | NULL | ref | camtimestamp,camname,CamNameCamTime | CamNameCamTime | 263 | INSIDE.camname,INSIDE.maxtimestamp | 1 | 100.00 | NULL |
| 2 | DERIVED | events | NULL | range | camname,CamNameCamTime | CamNameCamTime | 257 | NULL | 2 | 100.00 | Using index for group-by |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
3 rows in set, 1 warning (0.00 sec)
成功！
你能试试下面的 sql 吗？
SELECT OUTSIDE.owneruid
AS Owner,
OUTSIDE.propuid
AS Property,
OUTSIDE.camname
AS 'Camera Name',
OUTSIDE.direction
AS Direction,
OUTSIDE.camtimestamp
AS 'Event Time',
Convert_tz(Now(), 'UTC', 'US/Central')
AS Now,
Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') -
OUTSIDE.camtimestamp)
AS 'Elapsed Time'
FROM events OUTSIDE
WHERE OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
FROM events WHERE
events.camname = OUTSIDE.camname);
====================
这里是原来的提问者。我执行了上面的 SQL 但查询从未完成。我终于不得不按 CTRL C 来结束它。我重新检查了数据库连接并尝试了其他成功完成的通用查询并重试了上面的 SQL。它做了同样的事情。
这里是查询的解释：
mysql> explain SELECT OUTSIDE.owneruid
-> AS Owner,
-> OUTSIDE.propuid
-> AS Property,
-> OUTSIDE.camname
-> AS 'Camera Name',
-> OUTSIDE.direction
-> AS Direction,
-> OUTSIDE.camtimestamp
-> AS 'Event Time',
-> Convert_tz(Now(), 'UTC', 'US/Central')
-> AS Now,
-> Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') -
-> OUTSIDE.camtimestamp)
-> AS 'Elapsed Time'
-> FROM events OUTSIDE
->
-> WHERE OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
-> FROM events WHERE
-> events.camname = OUTSIDE.camname);
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
| 1 | PRIMARY | OUTSIDE | NULL | ALL | NULL | NULL | NULL | NULL | 264783 | 100.00 | Using where |
| 2 | DEPENDENT SUBQUERY | events | NULL | ref | camname,CamNameCamTime | CamNameCamTime | 257 | lpr.OUTSIDE.camname | 263842 | 100.00 | Using index |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
2 rows in set, 2 warnings (0.00 sec)
如果关于 (camname, camtimestamp) 的建议评论复合索引没有帮助，这可能是向您的数据库添加触发器的方案。如果您一直在寻找给定摄像机名称的最新时间戳，我会在摄像机名称 table 中添加一列，例如 LastEventID。然后，在您插入事件 table 期间，只需发出一个简化的
update CameraNameTable set
LastEventID = EventIDOfRecordJustInserted
where
CameraName = CameraNameOfEventInserted
那么，您的查询可以像
一样简单
select
E.*,
[your additional time, convert value columns]
from
CameraTable CT
JOIN Events E
on CT.LastEventID = E.ID
要在首次添加此新列后填写您的相机名称 table，您可以
Update CameraTable
JOIN ( select CamName, max( ID ) maxEventID
from Events
group by CamName ) PQ
on CameraTable.CamName = PQ.CamName
set LastEventID = PQ.maxEventID
您的触发器可能会为插入添加亚秒级计时，但即使您每天只做 2,000 次，也就是每分钟 4 次，每天 8 小时。我不认为这有什么重要意义。一旦插入下一个事件 ID 记录，您的 table 将始终更新，因此您的查询永远不需要继续应用 max() 并加入。您是根据一个确切的 ID 加入的，该 ID 将成为索引的一部分，并且只能 1:1 匹配。
MySQL 不能在查询中对 table 的每个引用使用多个索引，因此 camname 和 camtimestamp 上的两个简单索引可能仅在这种情况下用处有限。这样的索引非常适合选择一个日期范围内的记录，或者一个名字的所有记录；但是在搜索每个名字的最大值时，camname 索引的用处值得怀疑（因为无论如何它都必须检查每个名字的每个 camtimestamp），而 camstimestamp 仍然需要搜索每个记录以进行确保所有 camnames 都被考虑在内（最小的 camtimestamp 可能是特定 camname 的唯一时间戳）。
使用 (camname, camtimestamp) 上的复合索引，可以在子查询中快速识别每个 camname 的最大值，然后可以再次使用该索引来挑选出中的行table 匹配子查询的结果。
这是我的系统：
Linux Ubuntu 18.04 LTS
MySQL 5.7
我有一个查询（如下）需要很长时间才能完成。
完成需要 9 秒。在等待网页完成加载时，对于用户来说，这是一段难以承受的时间。
此外，数据集很大而且还在增长。事件 table 有 250,000 行。我倾向于每天添加 1200 到 1800 行。
为了帮助优化，我想添加索引等，但我不知道如何（以及是否）可以使用我的 derived/joined select 查询来做到这一点。
这是我的查询。 （有谁知道如何限制输出的宽度，这样我们就不必向右滚动就能看到整行了？）
mysql> explain select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname
AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 263103 | 100.00 | Using where |
| 1 | PRIMARY | OUTSIDE | NULL | ref | camtimestamp,camname | camtimestamp | 6 | INSIDE.maxtimestamp | 1 | 99.73 | Using where |
| 2 | DERIVED | events | NULL | index | camname | camname | 257 | NULL | 263103 | 100.00 | NULL |
+----+-------------+------------+------------+-------+----------------------+--------------+---------+---------------------+--------+----------+-------------+
3 rows in set, 1 warning (0.03 sec)
查询结果如下：
mysql> select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname
AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time | Now | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| 1 | 1 | wls1 | In | 2020-01-30 12:27:31 | 2020-01-30 12:29:53 | 00:03:42 |
| 1 | 1 | wls2 | Out | 2020-01-30 12:25:29 | 2020-01-30 12:29:53 | 00:07:04 |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (6.49 sec)
============================================= ============= 感谢所有回答问题或发表评论的人。我采纳了 Uueerdo 关于复合索引的建议，得出了以下结果：
mysql> create index CamNameCamTime on events (camname,camtimestamp);
mysql> select OUTSIDE.ownerUID as Owner,
OUTSIDE.propUID as Property,
OUTSIDE.camname as 'Camera Name',
OUTSIDE.direction as Direction,
OUTSIDE.camtimestamp as 'Event Time',
convert_tz(now(),'UTC','US/Central') as Now,
sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
from events OUTSIDE,
(select camname,max(camtimestamp) as maxtimestamp
from events group by camname) as INSIDE
where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| Owner | Property | Camera Name | Direction | Event Time | Now | Elapsed Time |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
| 1 | 1 | wls1 | In | 2020-01-30 18:43:19 | 2020-01-30 18:44:33 | 00:01:54 |
| 1 | 1 | wls2 | Out | 2020-01-30 18:41:51 | 2020-01-30 18:44:33 | 00:04:42 |
+-------+----------+-------------+-----------+---------------------+---------------------+--------------+
2 rows in set (0.00 sec)
mysql> explain select OUTSIDE.ownerUID as Owner,
-> OUTSIDE.propUID as Property,
-> OUTSIDE.camname as 'Camera Name',
-> OUTSIDE.direction as Direction,
-> OUTSIDE.camtimestamp as 'Event Time',
-> convert_tz(now(),'UTC','US/Central') as Now,
-> sec_to_time(convert_tz(now(),'UTC','US/Central') - OUTSIDE.CamTimeStamp) as 'Elapsed Time'
-> from events OUTSIDE,
-> (select camname,max(camtimestamp) as maxtimestamp
-> from events group by camname) as INSIDE
-> where OUTSIDE.camname = INSIDE.camname AND OUTSIDE.camtimestamp = INSIDE.maxtimestamp;
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | Using where |
| 1 | PRIMARY | OUTSIDE | NULL | ref | camtimestamp,camname,CamNameCamTime | CamNameCamTime | 263 | INSIDE.camname,INSIDE.maxtimestamp | 1 | 100.00 | NULL |
| 2 | DERIVED | events | NULL | range | camname,CamNameCamTime | CamNameCamTime | 257 | NULL | 2 | 100.00 | Using index for group-by |
+----+-------------+------------+------------+-------+-------------------------------------+----------------+---------+------------------------------------+------+----------+--------------------------+
3 rows in set, 1 warning (0.00 sec)
成功！
你能试试下面的 sql 吗？
SELECT OUTSIDE.owneruid
AS Owner,
OUTSIDE.propuid
AS Property,
OUTSIDE.camname
AS 'Camera Name',
OUTSIDE.direction
AS Direction,
OUTSIDE.camtimestamp
AS 'Event Time',
Convert_tz(Now(), 'UTC', 'US/Central')
AS Now,
Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') -
OUTSIDE.camtimestamp)
AS 'Elapsed Time'
FROM events OUTSIDE
WHERE OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
FROM events WHERE
events.camname = OUTSIDE.camname);
====================
这里是原来的提问者。我执行了上面的 SQL 但查询从未完成。我终于不得不按 CTRL C 来结束它。我重新检查了数据库连接并尝试了其他成功完成的通用查询并重试了上面的 SQL。它做了同样的事情。
这里是查询的解释：
mysql> explain SELECT OUTSIDE.owneruid
-> AS Owner,
-> OUTSIDE.propuid
-> AS Property,
-> OUTSIDE.camname
-> AS 'Camera Name',
-> OUTSIDE.direction
-> AS Direction,
-> OUTSIDE.camtimestamp
-> AS 'Event Time',
-> Convert_tz(Now(), 'UTC', 'US/Central')
-> AS Now,
-> Sec_to_time(Convert_tz(Now(), 'UTC', 'US/Central') -
-> OUTSIDE.camtimestamp)
-> AS 'Elapsed Time'
-> FROM events OUTSIDE
->
-> WHERE OUTSIDE.camtimestamp = (SELECT Max(camtimestamp)
-> FROM events WHERE
-> events.camname = OUTSIDE.camname);
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
| 1 | PRIMARY | OUTSIDE | NULL | ALL | NULL | NULL | NULL | NULL | 264783 | 100.00 | Using where |
| 2 | DEPENDENT SUBQUERY | events | NULL | ref | camname,CamNameCamTime | CamNameCamTime | 257 | lpr.OUTSIDE.camname | 263842 | 100.00 | Using index |
+----+--------------------+---------+------------+------+------------------------+----------------+---------+---------------------+--------+----------+-------------+
2 rows in set, 2 warnings (0.00 sec)
如果关于 (camname, camtimestamp) 的建议评论复合索引没有帮助，这可能是向您的数据库添加触发器的方案。如果您一直在寻找给定摄像机名称的最新时间戳，我会在摄像机名称 table 中添加一列，例如 LastEventID。然后，在您插入事件 table 期间，只需发出一个简化的
update CameraNameTable set
LastEventID = EventIDOfRecordJustInserted
where
CameraName = CameraNameOfEventInserted
那么，您的查询可以像
一样简单select
E.*,
[your additional time, convert value columns]
from
CameraTable CT
JOIN Events E
on CT.LastEventID = E.ID
要在首次添加此新列后填写您的相机名称 table，您可以
Update CameraTable
JOIN ( select CamName, max( ID ) maxEventID
from Events
group by CamName ) PQ
on CameraTable.CamName = PQ.CamName
set LastEventID = PQ.maxEventID
您的触发器可能会为插入添加亚秒级计时，但即使您每天只做 2,000 次，也就是每分钟 4 次，每天 8 小时。我不认为这有什么重要意义。一旦插入下一个事件 ID 记录，您的 table 将始终更新，因此您的查询永远不需要继续应用 max() 并加入。您是根据一个确切的 ID 加入的，该 ID 将成为索引的一部分，并且只能 1:1 匹配。
MySQL 不能在查询中对 table 的每个引用使用多个索引，因此 camname 和 camtimestamp 上的两个简单索引可能仅在这种情况下用处有限。这样的索引非常适合选择一个日期范围内的记录，或者一个名字的所有记录；但是在搜索每个名字的最大值时，camname 索引的用处值得怀疑（因为无论如何它都必须检查每个名字的每个 camtimestamp），而 camstimestamp 仍然需要搜索每个记录以进行确保所有 camnames 都被考虑在内（最小的 camtimestamp 可能是特定 camname 的唯一时间戳）。
使用 (camname, camtimestamp) 上的复合索引，可以在子查询中快速识别每个 camname 的最大值，然后可以再次使用该索引来挑选出中的行table 匹配子查询的结果。