MySQL 许多 (~10-100K) 短 select 查询的性能下降
MySQL degrading performance over many (~10-100K) short select queries
我目前正在使用 MySQL 数据库在 Node 上构建一个应用程序，在初始 运行 时我必须计算大约 250M 项。每个计算需要 3 SELECT 个语句。
我现在正在测试数千个，以了解影响性能的因素。我看到奇怪的行为。最初的计算 运行 非常快，不到一秒就处理完 1k。然后，在几千（~20k）之后 运行 它减慢到每 1k 大约 4 秒。
现在，如果我让数据库单独放置一段时间（比如说 1 分钟），它会恢复速度以进行前 ~10k 次计算。
我实现了并发限制，同时计算不超过 20 个。增加限制似乎会降低性能。我还没有完全测试这个参数如何影响计算速度。
似乎有某种缓冲区填满了，即使之前的所有 SELECT 都应该完成。
其中每个计算 2000 行：
Just calculations: 2.557s
Just calculations: 1.566s
Just calculations: 1.617s
Just calculations: 1.543s
Just calculations: 1.475s
Just calculations: 1.635s
Just calculations: 1.613s
Just calculations: 3.324s
Just calculations: 4.103s
Just calculations: 8.325s
Just calculations: 7.583s
Just calculations: 7.482s
Just calculations: 7.216s
Just calculations: 7.145s
Just calculations: 7.419s
Just calculations: 7.244s
Just calculations: 7.066s
Just calculations: 7.635s
Just calculations: 7.169s
Just calculations: 7.194s
可以清楚地看到性能下降。降级也会影响 INSERT 查询，所以我很确定它特别是数据库层。
如有任何关于提高性能的提示或信息，我们将不胜感激。
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| item | CREATE TABLE `item` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`sku` varchar(255) NOT NULL,
`base_unit_of_measure` varchar(255) DEFAULT NULL,
`unit_price` double NOT NULL,
`blocked` tinyint(1) NOT NULL,
`item_disc__group` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `item_sku_unique` (`sku`)
) ENGINE=InnoDB AUTO_INCREMENT=1375125 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| contact | CREATE TABLE `contact` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`contact_id` varchar(255) NOT NULL,
`client_id` varchar(255) DEFAULT NULL,
`customer_disc__group` int DEFAULT NULL,
`customer_price_group` int DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `contact_contact_id_unique` (`contact_id`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sales_price | CREATE TABLE `sales_price` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`sku` varchar(255) NOT NULL,
`sales_type` int NOT NULL,
`sales_code` varchar(255) DEFAULT NULL,
`unit_of_measure_code` varchar(255) DEFAULT NULL,
`minimum_quantity` int NOT NULL,
`unit_price` double NOT NULL,
`allow_line_disc` tinyint(1) NOT NULL,
`starting_date` date DEFAULT NULL,
`ending_date` date DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sales_price_unique` (`sku`,`sales_type`,`sales_code`,`starting_date`,`unit_of_measure_code`,`minimum_quantity`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sales_line_discount | CREATE TABLE `sales_line_discount` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`type` int NOT NULL,
`code` varchar(255) NOT NULL,
`sales_type` int NOT NULL,
`sales_code` varchar(255) DEFAULT NULL,
`unit_of_measure_code` varchar(50) DEFAULT NULL,
`minimum_quantity` int NOT NULL,
`line_discount` double NOT NULL,
`starting_date` date DEFAULT NULL,
`ending_date` date DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `line_discount_unique` (`type`,`code`,`sales_type`,`sales_code`,`starting_date`,`unit_of_measure_code`,`minimum_quantity`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
针对这些表的 运行 查询如下：
bindings: [ 'TESTCONTACT1', 1 ],
SELECT `contact`.*
FROM `contact`
WHERE `contact`.`contact_id` = ?
LIMIT ?
bindings: [
0, 'DUMMYSKU24999',
1, 'DUMMYGROUP',
'2021-08-30', '2021-08-30',
0, 'TESTCONTACT1',
1, 2,
2, 3,
'TESTCAMP1', 'TESTCAMP2'
]
SELECT `sales_line_discount`.*
FROM `sales_line_discount`
WHERE ( ( `type` = ?
AND `code` = ?
OR ( `type` = ?
AND `code` = ? ) )
AND `starting_date` <= ?
AND `ending_date` >= ? )
AND ( `sales_type` = ?
AND `sales_code` = ?
OR ( `sales_type` = ?
AND `sales_code` = ? )
OR ( `sales_type` = ? )
OR ( `sales_type` = ?
AND `sales_code` IN ( ?, ? ) ) )
bindings: [
'DUMMYSKU24999', '2021-08-30',
'2021-08-30', 0,
'TESTCONTACT1', 1,
2, 2,
3, 'TESTCAMP1',
'TESTCAMP2'
],
SELECT `sales_price`.*
FROM `sales_price`
WHERE `sku` = ?
AND `starting_date` <= ?
AND `ending_date` >= ?
AND ( `sales_type` = ?
AND `sales_code` = ?
OR ( `sales_type` = ?
AND `sales_code` = ? )
OR ( `sales_type` = ? )
OR ( `sales_type` = ?
AND `sales_code` IN ( ?, ? ) ) )
最后的原因是后台进程周期性地占用应用的部分可用资源。
由于计算机一点也不落后，我曾假设它只使用一个内核并且在其他内核上工作不会影响它，但 htop 证明不同。杀死所有不必要的进程解决了性能问题。
我目前正在使用 MySQL 数据库在 Node 上构建一个应用程序，在初始 运行 时我必须计算大约 250M 项。每个计算需要 3 SELECT 个语句。 我现在正在测试数千个，以了解影响性能的因素。我看到奇怪的行为。最初的计算 运行 非常快，不到一秒就处理完 1k。然后，在几千（~20k）之后 运行 它减慢到每 1k 大约 4 秒。
现在，如果我让数据库单独放置一段时间（比如说 1 分钟），它会恢复速度以进行前 ~10k 次计算。
我实现了并发限制，同时计算不超过 20 个。增加限制似乎会降低性能。我还没有完全测试这个参数如何影响计算速度。 似乎有某种缓冲区填满了，即使之前的所有 SELECT 都应该完成。
其中每个计算 2000 行：
Just calculations: 2.557s
Just calculations: 1.566s
Just calculations: 1.617s
Just calculations: 1.543s
Just calculations: 1.475s
Just calculations: 1.635s
Just calculations: 1.613s
Just calculations: 3.324s
Just calculations: 4.103s
Just calculations: 8.325s
Just calculations: 7.583s
Just calculations: 7.482s
Just calculations: 7.216s
Just calculations: 7.145s
Just calculations: 7.419s
Just calculations: 7.244s
Just calculations: 7.066s
Just calculations: 7.635s
Just calculations: 7.169s
Just calculations: 7.194s
可以清楚地看到性能下降。降级也会影响 INSERT 查询，所以我很确定它特别是数据库层。
如有任何关于提高性能的提示或信息，我们将不胜感激。
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| item | CREATE TABLE `item` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`sku` varchar(255) NOT NULL,
`base_unit_of_measure` varchar(255) DEFAULT NULL,
`unit_price` double NOT NULL,
`blocked` tinyint(1) NOT NULL,
`item_disc__group` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `item_sku_unique` (`sku`)
) ENGINE=InnoDB AUTO_INCREMENT=1375125 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| contact | CREATE TABLE `contact` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`contact_id` varchar(255) NOT NULL,
`client_id` varchar(255) DEFAULT NULL,
`customer_disc__group` int DEFAULT NULL,
`customer_price_group` int DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `contact_contact_id_unique` (`contact_id`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sales_price | CREATE TABLE `sales_price` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`sku` varchar(255) NOT NULL,
`sales_type` int NOT NULL,
`sales_code` varchar(255) DEFAULT NULL,
`unit_of_measure_code` varchar(255) DEFAULT NULL,
`minimum_quantity` int NOT NULL,
`unit_price` double NOT NULL,
`allow_line_disc` tinyint(1) NOT NULL,
`starting_date` date DEFAULT NULL,
`ending_date` date DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `sales_price_unique` (`sku`,`sales_type`,`sales_code`,`starting_date`,`unit_of_measure_code`,`minimum_quantity`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sales_line_discount | CREATE TABLE `sales_line_discount` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`type` int NOT NULL,
`code` varchar(255) NOT NULL,
`sales_type` int NOT NULL,
`sales_code` varchar(255) DEFAULT NULL,
`unit_of_measure_code` varchar(50) DEFAULT NULL,
`minimum_quantity` int NOT NULL,
`line_discount` double NOT NULL,
`starting_date` date DEFAULT NULL,
`ending_date` date DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `line_discount_unique` (`type`,`code`,`sales_type`,`sales_code`,`starting_date`,`unit_of_measure_code`,`minimum_quantity`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
针对这些表的 运行 查询如下：
bindings: [ 'TESTCONTACT1', 1 ],
SELECT `contact`.*
FROM `contact`
WHERE `contact`.`contact_id` = ?
LIMIT ?
bindings: [
0, 'DUMMYSKU24999',
1, 'DUMMYGROUP',
'2021-08-30', '2021-08-30',
0, 'TESTCONTACT1',
1, 2,
2, 3,
'TESTCAMP1', 'TESTCAMP2'
]
SELECT `sales_line_discount`.*
FROM `sales_line_discount`
WHERE ( ( `type` = ?
AND `code` = ?
OR ( `type` = ?
AND `code` = ? ) )
AND `starting_date` <= ?
AND `ending_date` >= ? )
AND ( `sales_type` = ?
AND `sales_code` = ?
OR ( `sales_type` = ?
AND `sales_code` = ? )
OR ( `sales_type` = ? )
OR ( `sales_type` = ?
AND `sales_code` IN ( ?, ? ) ) )
bindings: [
'DUMMYSKU24999', '2021-08-30',
'2021-08-30', 0,
'TESTCONTACT1', 1,
2, 2,
3, 'TESTCAMP1',
'TESTCAMP2'
],
SELECT `sales_price`.*
FROM `sales_price`
WHERE `sku` = ?
AND `starting_date` <= ?
AND `ending_date` >= ?
AND ( `sales_type` = ?
AND `sales_code` = ?
OR ( `sales_type` = ?
AND `sales_code` = ? )
OR ( `sales_type` = ? )
OR ( `sales_type` = ?
AND `sales_code` IN ( ?, ? ) ) )
最后的原因是后台进程周期性地占用应用的部分可用资源。
由于计算机一点也不落后，我曾假设它只使用一个内核并且在其他内核上工作不会影响它，但 htop 证明不同。杀死所有不必要的进程解决了性能问题。