优化查询,多表连接

Optimizing query, join of many tables

我有一个 table,我需要在其中连接很多不同的 table。数据集有 140 000 条记录。

示例如下:

SELECT SQL_CALC_FOUND_ROWS e.designation
                         , e.remark
                         , e.moment
                         , e.rpm
                         , e.cycleK
                         , c.type
                         , d.description
                         , a.PAnr
                         , b.family
                         , b.articlenrKronhjul
                         , b.ratio
                         , a.oiltype
                         , a.oiltemp
                         , a.createdBy
                         , a.createdDate
                      FROM testdata_test a
                         , testdata_gear b
                         , testdata_damcategory c
                         , testdata_damage d
                         , testdata_result e
                     WHERE a.id = e.test_id 
                       AND e.id = d.result_id 
                       AND a.id = b.test_id 
                       AND c.id = d.category_id
                     ORDER 
                        BY designation asc
                     LIMIT 0, 10

平均耗时约 1 秒,如何加快速度? 我一直在尝试在某些列上添加一些索引,但没有太大改进。

有人有什么建议吗?

编辑:

这是我的常规和 JSON 格式的查询计划:

+----+-------------+-------+------+---------------------------------------------------+--------------------------+---------+----------------+------+----------------------------------------------------+
| id | select_type | table | type | possible_keys                                     | key                      | key_len | ref            | rows | Extra                                              |
+----+-------------+-------+------+---------------------------------------------------+--------------------------+---------+----------------+------+----------------------------------------------------+
|  1 | SIMPLE      | a     | ALL  | PRIMARY                                           | NULL                     | NULL    | NULL           | 10617 | Using where; Using temporary; Using filesort      |
|  1 | SIMPLE      | b     | ref  | TestData_gear_2e06cda4                            | TestData_gear_2e06cda4   | 5       | webappdev.a.id |     1 | NULL                                              |
|  1 | SIMPLE      | e     | ref  | PRIMARY,TestData_result_2e06cda4                  | TestData_result_2e06cda4 | 4       | webappdev.a.id |     5 | NULL                                              |
|  1 | SIMPLE      | d     | ref  | TestData_damage_b583a629,TestData_damage_57f06544 | TestData_damage_57f06544 | 4       | webappdev.e.id |     1 | NULL                                              |
|  1 | SIMPLE      | c     | ALL  | PRIMARY                                           | NULL                     | NULL    | NULL           |     4 | Using where; Using join buffer (Block Nested Loop)|
+----+-------------+-------+------+---------------------------------------------------+--------------------------+---------+----------------+-------+---------------------------------------------------+
5 rows in set (0.00 sec)

| {
  "query_block": {
    "select_id": 1,
    "ordering_operation": {
      "using_temporary_table": true,
      "using_filesort": true,
      "nested_loop": [
        {
          "table": {
            "table_name": "a",
            "access_type": "ALL",
            "possible_keys": [
              "PRIMARY"
            ],
            "rows": 10617,
            "filtered": 100,
            "attached_condition": "(`webappdev`.`a`.`id` is not null)"
          }
        },
        {
          "table": {
            "table_name": "b",
            "access_type": "ref",
            "possible_keys": [
              "TestData_gear_2e06cda4"
            ],
            "key": "TestData_gear_2e06cda4",
            "used_key_parts": [
              "test_id"
            ],
            "key_length": "5",
            "ref": [
              "webappdev.a.id"
            ],
            "rows": 1,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "e",
            "access_type": "ref",
            "possible_keys": [
              "PRIMARY",
              "TestData_result_2e06cda4"
            ],
            "key": "TestData_result_2e06cda4",
            "used_key_parts": [
              "test_id"
            ],
            "key_length": "4",
            "ref": [
              "webappdev.a.id"
            ],
            "rows": 5,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "d",
            "access_type": "ref",
            "possible_keys": [
              "TestData_damage_b583a629",
              "TestData_damage_57f06544"
            ],
            "key": "TestData_damage_57f06544",
            "used_key_parts": [
              "result_id"
            ],
            "key_length": "4",
            "ref": [
              "webappdev.e.id"
            ],
            "rows": 1,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "c",
            "access_type": "ALL",
            "possible_keys": [
              "PRIMARY"
            ],
            "rows": 4,
            "filtered": 75,
            "using_join_buffer": "Block Nested Loop",
            "attached_condition": "(`webappdev`.`c`.`id` = `webappdev`.`d`.`cate
gory_id`)"
          }
        }
      ]
    }
  }
} |

这是我的 CREATE TABLES 和 tables

的计数
mysql> SHOW CREATE TABLE testdata_test;
| testdata_test | CREATE TABLE `testdata_test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `PAnr` int(11) NOT NULL,
  `projectAcc` varchar(20) DEFAULT NULL,
  `reportnr` varchar(20) DEFAULT NULL,
  `oiltype` varchar(40) DEFAULT NULL,
  `oiltemp` int(11) DEFAULT NULL,
  `headline1` varchar(40) DEFAULT NULL,
  `headline2` varchar(40) DEFAULT NULL,
  `testDescription` longtext,
  `TestName` varchar(9) DEFAULT NULL,
  `createdBy` varchar(6) NOT NULL,
  `createdDate` date NOT NULL,
  PRIMARY KEY (`id`),
  FULLTEXT KEY `test_index_testdescription` (`testDescription`)
) ENGINE=InnoDB AUTO_INCREMENT=14172 DEFAULT CHARSET=utf8 |
1 row in set (0.00 sec)
    mysql> SELECT count(*) FROM testdata_test;
+----------+
| count(*) |
+----------+
|    14161 |
+----------+
1 row in set (0.01 sec)





mysql> SHOW CREATE TABLE testdata_gear;
| testdata_gear | CREATE TABLE `testdata_gear` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `family` varchar(20) NOT NULL,
  `articlenrKronhjul` int(11) DEFAULT NULL,
  `revisionK` varchar(200) DEFAULT NULL,
  `articlenrPinjong` int(11) DEFAULT NULL,
  `revisionP` varchar(200) DEFAULT NULL,
  `ratio` double DEFAULT NULL,
  `geardata` varchar(100) DEFAULT NULL,
  `remark` varchar(40) DEFAULT NULL,
  `test_id` int(11),
  PRIMARY KEY (`id`),
  KEY `TestData_gear_2e06cda4` (`test_id`),
  CONSTRAINT `TestData_gear_test_id_325c2ab6_fk_TestData_test_id` FOREIGN KEY (`
test_id`) REFERENCES `testdata_test` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=14167 DEFAULT CHARSET=utf8 |

mysql> SELECT count(*) FROM testdata_gear;

+----------+
| count(*) |
+----------+
|    14157 |
+----------+
1 row in set (0.01 sec)

  | testdata_result | CREATE TABLE `testdata_result` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `designation` varchar(20) NOT NULL,
  `remark` varchar(200) DEFAULT NULL,
  `moment` double DEFAULT NULL,
  `rpm` int(11) DEFAULT NULL,
  `cycleK` int(11) DEFAULT NULL,
  `test_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `test_id_result_index` (`test_id`) USING BTREE,
  KEY `result_designation_index` (`designation`) USING BTREE,
  CONSTRAINT `TestData_result_test_id_5ed0cbc8_fk_TestData_test_id` FOREIGN KEY
(`test_id`) REFERENCES `testdata_test` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=141382 DEFAULT CHARSET=utf8 |
mysql> SELECT count(*) FROM testdata_result;
+----------+
| count(*) |
+----------+
|   141323 |
+----------+
1 row in set (0.03 sec)

mysql> SHOW CREATE TABLE testdata_damage;
| testdata_damage | CREATE TABLE `testdata_damage` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `description` longtext NOT NULL,
  `part` varchar(100) DEFAULT NULL,
  `timestamp` datetime(6) NOT NULL,
  `category_id` int(11),
  `result_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `TestData_damage_b583a629` (`category_id`),
  KEY `TestData_damage_57f06544` (`result_id`),
  FULLTEXT KEY `damage_index_description` (`description`),
  CONSTRAINT `TestData_damage_category_id_215346e4_fk_TestData_damcategory_id` F
OREIGN KEY (`category_id`) REFERENCES `testdata_damcategory` (`id`),
  CONSTRAINT `TestData_damage_result_id_2fb199b2_fk_TestData_result_id` FOREIGN
KEY (`result_id`) REFERENCES `testdata_result` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=141341 DEFAULT CHARSET=utf8 |

mysql> SELECT count(*) FROM testdata_damage;
+----------+
| count(*) |
+----------+
|   141291 |
+----------+
1 row in set (0.04 sec)

mysql> SHOW CREATE TABLE testdata_damcategory;
| testdata_damcategory | CREATE TABLE `testdata_damcategory` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `type` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8 |

mysql> SELECT count(*) FROM testdata_damcategory;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)


SELECT COUNT(*) FROM (
SELECT e.designation
...
) AS x;

+----------+
| COUNT(*) |
+----------+
|   141298 |
+----------+
1 row in set (2.40 sec)¨

编辑: 附指定索引说明

| {
  "query_block": {
    "select_id": 1,
    "ordering_operation": {
      "using_temporary_table": true,
      "using_filesort": true,
      "nested_loop": [
        {
          "table": {
            "table_name": "a",
            "access_type": "ALL",
            "possible_keys": [
              "PRIMARY"
            ],
            "rows": 10617,
            "filtered": 100,
            "attached_condition": "(`webappdev`.`a`.`id` is not null)"
          }
        },
        {
          "table": {
            "table_name": "b",
            "access_type": "ref",
            "possible_keys": [
              "TestData_gear_2e06cda4",
              "test_id_gear_index"
            ],
            "key": "TestData_gear_2e06cda4",
            "used_key_parts": [
              "test_id"
            ],
            "key_length": "5",
            "ref": [
              "webappdev.a.id"
            ],
            "rows": 1,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "e",
            "access_type": "ref",
            "possible_keys": [
              "PRIMARY",
              "test_id_result_index"
            ],
            "key": "test_id_result_index",
            "used_key_parts": [
              "test_id"
            ],
            "key_length": "4",
            "ref": [
              "webappdev.a.id"
            ],
            "rows": 4,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "d",
            "access_type": "ref",
            "possible_keys": [
              "TestData_damage_b583a629",
              "TestData_damage_57f06544",
              "result_id_damage_index"
            ],
            "key": "TestData_damage_57f06544",
            "used_key_parts": [
              "result_id"
            ],
            "key_length": "4",
            "ref": [
              "webappdev.e.id"
            ],
            "rows": 1,
            "filtered": 100
          }
        },
        {
          "table": {
            "table_name": "c",
            "access_type": "ALL",
            "possible_keys": [
              "PRIMARY"
            ],
            "rows": 4,
            "filtered": 75,
            "using_join_buffer": "Block Nested Loop",
            "attached_condition": "(`webappdev`.`c`.`id` = `webappdev`.`d`.`cate
gory_id`)"
          }
        }
      ]
    }
  }
} |

最终更新: 这是最终成功的查询,将我的查询时间降低到 0.1 秒以下。

SELECT e.designation
    , e.remark
    , e.moment
    , e.rpm
    , e.cycleK
    , c.type
    , d.description
    , a.PAnr
    , b.family
    , b.articlenrKronhjul
    , b.ratio
    , a.oiltype
    , a.oiltemp
    , a.createdBy
    , a.createdDate
FROM (
     SELECT e.id, e.designation, e.remark, e.moment, e.rpm, e.cycleK, e.test_id
         FROM testdata_result e
         ORDER BY moment asc
         LIMIT 10
      )e
JOIN testdata_damage AS d  ON d.result_id = e.id
JOIN testdata_test AS a ON a.id = e.test_id
JOIN testdata_gear AS b ON a.id = b.test_id
JOIN testdata_damcategory as c ON c.id = d.category_id;
INDEX(designation)

可能会有所帮助。

没有 LIMIT 10 你得到多少行?

请为每个 table 提供 SHOW CREATE TABLE,以便我们进一步讨论。

编辑

没关系。 141K 行的结果集,包括 LONGTEXT 将需要时间。你很幸运,只需要一秒钟左右。有很多东西要铲。磁盘走得这么快;网络只有这么快。导致它成为 "slow" 的不是查询或索引;这是请求。

您可以用 141K 行做什么?当然,您不会将所有这些都显示给用户。如果你正在以某种方式咀嚼它们,那肯定需要几秒钟,这意味着查询只是总时间的一部分。

我建议使用索引以避免执行文件排序——但我没有注意到它是 LONGTEXT,因此使用常规索引无效。 FULLTEXT 索引没有排序。底线:我的 INDEX(designation) 没用。

基本上没有办法加快查询速度目前。如果您需要一次获取 10 行,可能有一些技巧(不涉及 OFFSET)可以使它更好地工作。

乍一看,我认为 SQL_CALC_FOUND_ROWS 正在阻止任何加速查询的尝试。这要求所有 JOINs 都完成。在我之前的回答中,我希望在 10 行后短路评估。那是不可能的。

因此,"right" 解决方案涉及更改应用程序以执行有关计数的操作。

  • 摆脱它(并说服用户他们并不真正需要它);
  • 将其缓存在某处(每晚?);
  • 近似值

如果在删除 SQL_CALC_FOUND_ROWS 后,您仍然无法优化,请执行以下操作:

SELECT ...
    FROM (
        SELECT e.id, e.remark, e.moment, e.rpm, e.cycleK
            FROM result
            ORDER BY designation  -- check that there is a plain INDEX on this
            LIMIT 10
         ) ON e
    JOIN damage AS d  ON d.result_id = e.id  -- check for INDEX(result_id)
    JOIN ... (the rest of the tables other than `e`)
    ORDER BY designation  -- yes, this redundancy is necessary

此编码 'trick' 是将 LIMIT(或 GROUP BY)移动到子查询中以减少在执行 JOINs 时要处理的行数。这不是 JOINing 很多行,然后扔掉其中的大部分。