提高复杂 SQL 查询的性能

Improve performance of complex SQL query

几个月以来,我一直在从公开来源收集足球比赛的数据。为此,我通过 XPath 使用 PHP 获取特定网站 URL 的数据,该网站显示特定足球比赛的数据。然后我进行一些数据编辑,使它们符合我需要的方式。下一步也是最后一步是在 table 秒内将它们传输到我的 MySQL 数据库中。

随着数据库的快速增长,我慢慢地 运行 遇到了严重的性能问题。因为我在我的电脑上本地做所有事情而且那台机器不是怪物,所以处理一场比赛已经需要一些时间。感受一下这有多快:在数据挖掘的最初几天,一场比赛大约需要 24 秒。但是,现在平均值超过了 60 秒的阈值。

到目前为止,我偶尔会研究 PHP 代码并尝试在可能的情况下对其进行改进,因为我认为主要问题在于不太干净的代码片段。虽然它有点帮助,但平均时间在几天后进一步增加,最近我开始意识到必须有另一个耗时的问题。所以我制作了一个测试 PHP 脚本,它在 运行 运行主代码时执行某种日志记录。

它表明我为将数据插入数据库 table 所做的一些 SQL 查询平均花费大量时间(我在这里分析了 100 个匹配项):

再次检查查询,我发现它们非常复杂。

涉及的 table 个:

tblStartingSquad

+----+---------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+--------+  
| id | matchID | player1ID | player2ID | player3ID | player4ID | player5ID | player6ID | player7ID | player8ID | player9ID | player10ID | player11ID | clubID |
+----+---------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+--------+
| 1  |    1    |     1     |     2     |     3     |     4     |     5     |     6     |     7     |     8     |     9     |     10     |     11     |   1    |
| 2  |    1    |    12     |    13     |    14     |    15     |    16     |    17     |    16     |    17     |    18     |     19     |     20     |   2    |
| 3  |    2    |     1     |     2     |     3     |     4     |     5     |     6     |     7     |     8     |     9     |     10     |     11     |   1    |
| 4  |    2    |    21     |    22     |    23     |    24     |    25     |    26     |    27     |    28     |    29     |     30     |     31     |   3    |
+----+---------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+--------+

tblSubstitutes

+----+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+--------+  
| id | matchID | player12ID | player13ID | player14ID | player15ID | player16ID | player17ID | player18ID | player19ID | player20ID | player21ID | player22ID | player23ID | clubID |
+----+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+--------+
| 1  |    1    |     32     |     33     |     34     |     35     |     36     |     37     |     38     |     39     |     40     |     41     |     42     |     43     |   1    |
| 2  |    1    |     44     |     45     |     46     |     47     |     48     |     49     |     50     |     51     |     52     |     53     |     54     |     55     |   2    |
| 3  |    2    |     32     |     33     |     34     |     35     |     36     |     37     |     38     |     39     |     40     |     41     |     42     |     43     |   1    |
| 4  |    2    |     56     |     57     |     58     |     59     |     60     |     61     |     61     |     62     |     63     |     64     |     65     |     66     |   3    |
+----+---------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+--------+

tblMatch

+---------+---------------------+-------------+------------------+
| matchID |         date        |    coach1   |      coach2      |
+---------+---------------------+-------------+------------------+
|    1    | 2006-08-19 22:00:00 | Piotr Nowak | Fernando Clavijo |
|    2    | 2006-08-15 21:00:00 | Piotr Nowak |   Mustafa Ugur   |
+---------+---------------------+-------------+------------------+

tblPlayer

+----------+------------------------+------------------+
| playerID |       namePlayer       |      short       |
+----------+------------------------+------------------+
|     1    |       Enis Ulusan      |    enis-ulusan   |
|     2    |   Grant Robert Murray  |   grant-murray   |
|     3    |     Evgeniy Shpedt     |  evgeniy-shpedt  |
|     4    | Mihai Alexandru Costea |   mihai-costea   |
|     5    |       Jan Zolna        |     jan-zolna    |
|     6    |    Adrian Gheorghiu    | adrian-gheorghiu |
|     7    | Marius Marian Croitoru | marius-croitoru  |
|     8    |    Jacov Nachtailer    | jacov-nachtailer |
|    ...   |          ...           |        ...       |
+----------+------------------------+------------------+

tblClub

+--------+-----------------+
| clubID |    nameClub     |
+--------+-----------------+
|    1   |   D.C. United   |
|    2   | Colorado Rapids |
|    3   | Caykur Rizespor |
+--------+-----------------+

这些是涉及的查询:

SQL查询首发阵容

$tblstarting_squad = 'INSERT INTO tblStartingSquad (matchID, player1ID, player2ID, player3ID, player4ID, player5ID, player6ID, player7ID, player8ID, player9ID, player10ID, player11ID, clubID) 
                    SELECT
                        (SELECT matchID FROM tblMatch WHERE date = "' . $match_date . '" AND coach1 = "' . $match_coach_home . '" AND coach2 = "' . $match_coach_away . '"), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[0] . '" AND short = "' . $player_short[0] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[1] . '" AND short = "' . $player_short[1] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[2] . '" AND short = "' . $player_short[2] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[3] . '" AND short = "' . $player_short[3] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[4] . '" AND short = "' . $player_short[4] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[5] . '" AND short = "' . $player_short[5] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[6] . '" AND short = "' . $player_short[6] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[7] . '" AND short = "' . $player_short[7] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[8] . '" AND short = "' . $player_short[8] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[9] . '" AND short = "' . $player_short[9] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[10] . '" AND short = "' . $player_short[10] . '" LIMIT 1), 
                        (SELECT clubID FROM tblClub WHERE nameClub = "' . $match_club[1] . '" LIMIT 1)
                    WHERE NOT EXISTS (
                        SELECT e.matchID 
                        FROM tblStartingSquad As e
                        INNER JOIN tblMatch As m
                            ON e.matchID = m.matchID
                        WHERE m.date = "' . $match_date . '" AND m.coach1 = "' . $match_coach_home . '" AND m.coach2 = "' . $match_coach_away . '" AND e.clubID = (SELECT clubID FROM tblClub WHERE nameClub = "' . $match_club[1] . '")
                    );';

if (!mysqli_query($db_connection, $tblstarting_squad)) {
                    echo("Error description $tblstarting_squad: " . mysqli_error($db_connection) . "<br />");
                }

SQL 查询替补球员

$tblsubstitutes = 'INSERT INTO tblSubstitutes (matchID, player12ID, player13ID, player14ID, player15ID, player16ID, player17ID, player18ID, player19ID, player20ID, player21ID, player22ID, player23ID, clubID) 
                    SELECT
                        (SELECT matchID FROM tblMatch WHERE date = "' . $match_date . '" AND coach1 = "' . $match_coach_home . '" AND coach2 = "' . $match_coach_away . '"), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[11] . '" AND short = "' . $player_short[11] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[12] . '" AND short = "' . $player_short[12] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[13] . '" AND short = "' . $player_short[13] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[14] . '" AND short = "' . $player_short[14] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[15] . '" AND short = "' . $player_short[15] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[16] . '" AND short = "' . $player_short[16] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[17] . '" AND short = "' . $player_short[17] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[18] . '" AND short = "' . $player_short[18] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[19] . '" AND short = "' . $player_short[19] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[20] . '" AND short = "' . $player_short[20] . '" LIMIT 1), 
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[21] . '" AND short = "' . $player_short[21] . '" LIMIT 1),
                        (SELECT playerID FROM tblPlayer WHERE namePlayer = "' . $player_name[22] . '" AND short = "' . $player_short[22] . '" LIMIT 1), 
                        (SELECT clubID FROM tblClub WHERE nameClub = "' . $match_club[1] . '" LIMIT 1)
                    WHERE NOT EXISTS (
                        SELECT e.matchID 
                        FROM tblSubstitutes As e
                        INNER JOIN tblMatch As m
                            ON e.matchID = m.matchID
                        WHERE m.date = "' . $match_date . '" AND m.coach1 = "' . $match_coach_home . '" AND m.coach2 = "' . $match_coach_away . '" AND e.clubID = (SELECT clubID FROM tblClub WHERE nameClub = "' . $match_club[1] . '")
                    );';

if (!mysqli_query($db_connection, $tblsubstitutes)) {
                    echo("Error description $tblsubstitutes: " . mysqli_error($db_connection) . "<br />");
                }

这两个查询实际上是相同的。如果没有其他条目具有相同的数据,他们将 11 个(分别为 12 个)玩家的 playerID 插入 tblStartingSquad(分别为 tblSubstitutes)。 playerID 必须事先在数据库中检查,因为原始数据没有个人 ID。通过 namePlayershort 从 table tblPlayer.

选择它

tables tblStartingSquadtblSubstitutes 本身目前包含 110,000 行(55,000 个匹配),tblPlayer 是 100,000 行。

我在谷歌上搜索了一些解决方案,但没能找到任何可以提高整体速度的方法。我理解的一个问题是我必须单独检查每个玩家,所以我得到 11 和 12 个子查询。这不是很优雅,但我真的不知道如何改进它。也许 Whosebug 上有人有建议?

正如 O.Jones 在评论中所述,查看更多 (php) 代码有助于判断性能问题。 尽管重新设计了您的数据库,但另一个快速建议实际上是 运行 一个循环并使用 prepared statement 来对玩家 ID 进行单独查询。这可能会给您带来轻微的性能提升。

在我看来确实合乎逻辑,在 PHP 中做更多而不是将数据获取逻辑外包给 SQL。

重新考虑您的 wide table 设计以实现 long table 设计。带编号后缀的列从来都不是理想的数据存储。行很便宜。色谱柱很贵。连接、聚合、搜索、索引等在长格式中要容易得多。否则,您的查询将很复杂,因为您显示有 12 个子查询甚至自连接!

有趣的是,您的 tblClubtblPlayer 是长格式但 tblStartingSquadtblSubstitutes!简单地说,将所有无关的玩家列删除到 one 中,其中行表示不同的玩家:

tblStartingSquad

 ID   MatchID   PlayerID    ClubID  
  1         1          5         1
  2         1          8         1
  3         1          9         1
...

tblSubstitutes

 ID   MatchID   PlayerID    ClubID
  1         1          2         1
  2         1         16         1
  3         1          7         1
...

tblMatch (为清楚起见重命名了教练列)

ID                    Date      HomeCoach          AwayCoach
 1     2006-08-19 22:00:00    Piotr Nowak   Fernando Clavijo
 2     2006-08-15 21:00:00    Piotr Nowak       Mustafa Ugur

PHP

从这个数据库设计中,您可以 运行 一个更简单的 PHP 参数化查询调用,使用 PDO 更容易,而不是 mysqli 来绑定来自数组。

// OPEN CONNECTION
$dbconn = new PDO("mysql:host=$servername;dbname=$dbname", $username, $password);
// SET PDO ERROR MODE TO EXCEPTION
$dbconn -> setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

// PREPARED STATEMENT
$sql = "INSERT INTO tblStartingSquad (`Match`, `PlayerID`, `ClubID`)
        SELECT m.MatchID, p1.PlayerID, c.ClubID
        FROM 
           (SELECT p.PlayerID
            FROM tblPlayer p
            WHERE p.namePlayer IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
           ) p1
        INNER JOIN 
           (SELECT p.PlayerID
            FROM tblPlayer p
            WHERE p.short IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
           ) p2 ON p1.PlayerID = p2.PlayerID
        CROSS JOIN
            (SELECT MatchID 
             FROM tblMatch
             WHERE `date` = ? AND HomeCoach = ? AND AwayCoach = ?) m
        CROSS JOIN
            (SELECT ClubID 
             FROM tblClub
             WHERE nameClub = ?) c
        WHERE NOT EXISTS
             (SELECT 1 FROM tblStartingSquad As e
              WHERE e.MatchID = m.matchID)"

try {
     // INITIALIZE STATEMENT
     $stmt = $dbconn->prepare($sql);

     $params = array($player_name[0], $player_name[1], $player_name[2], 
                     $player_name[3], $player_name[4], $player_name[5], 
                     $player_name[6], $player_name[7], $player_name[8], 
                     $player_name[9], $player_name[10],  
                     $player_short[0], $player_short[1], $player_short[2],
                     $player_short[3], $player_short[4], $player_short[5], 
                     $player_short[6], $player_short[7], $player_short[8], 
                     $player_short[9], $player_short[10],         
                     $match_date, $match_coach_home, $match_coach_away, $match_club);

     // ITERATIVELY BIND PARAMS
     foreach($params as $key => $val) {
        $stmt->bindParam($key+1, $val, PDO::PARAM_STR);
     }

     // EXECUTE ACTION
     $stmt->execute();

} catch (PDOException $e) {
     echo "Error: " . $e->getMessage();
}

tblSubstitutes 进行类似的调用,调整追加查询的目标和 WHERE 子句和参数值。

语句并不复杂。它们只包含大量查找。因此,请确保查找速度很快。您需要以下索引。如果您还没有它们,请将它们添加到您的数据库中。

create index idx_find_player on tblplayer (nameplayer, short, playerid);
create index idx_find_club on tblclub (nameclub, clubid)
create index idx_find_match on tblmatch (date, coach1, coach2, matchid)

create index idx_find_squad1 on tblstartingsquad (matchid, clubid)
create index idx_find_squad2 on tblSsartingsquad (clubid, matchid)

create index idx_find_subs1 on tblsubstitutes (matchid, clubid)
create index idx_find_subs2 on tblsubstitutes (clubid, matchid)

我不确定哪个小队索引更有可能被使用,所以创建两个并查看 DBMS 选择哪个。然后你可以放下另一个。替代索引相同。

为了获得准确的答案,我们必须在 SQL 中查看您的执行计划,将其发送至此处,以便我可以帮助您解决问题

在那之前,我认为你做错了,而不是为单行写一个 select 你可以简单地在你的数据库中定义用户定义的 - table 并传递你的值。通过这样做你可以在你的 SQL 和你的服务器端代码中有更好的性能,我向你保证,你在第一级和之后有最好的性能,正如我之前所说的需要执行计划

  1. 声明你的类型(不要忘记改变你自己的数据类型)

创建类型 [dbo].[com_ListOfGuid] AS TABLE(NamePlayer NVARCHAR(256) NOT NULL,Short NVARCHAR(256) NOT NULL)

  1. 而不是在代码中创建 select 语句,只需声明一个数据 table 并将数据填充到其中并像其他参数一样传递它(不要忘记设置你的列名与你的用户定义的-table-type 完全一样) 3.in 你的 SQL 代码只是加入用户定义的-table-类型和你的 table 并将它们插入你的目标 table (我给你一个例子和你随着你的需要而改变)

    创建过程Procedure_Your_Name @ObjectTable Type_Your_Name 只读 作为 插入 TARGET_Table () SELECT * 从 first_Table F 内连接 @ObjectTable T ON F.NamePlayer = t.NamePlayer 和 t.Short = f.Short