如何识别 MySQL 查询的时间非线性增加?

How do I identify non-linear increase in time for MySQL query?

下面是我运行宁的两个查询。一个需要 75-80 秒,一个需要 1.0-1.5 秒。这两个结果都显示了预期的 50 行 channel_administrators.channel_partner_ids。更快查询和更慢查询之间的区别在于 SELECT 从登录名 table 中选择唯一登录名。登录 table 有 460833 行,我知道这会减慢查询速度。我发现这出乎意料的原因是,当 运行 将此代码单独放在一个 channel_administrators.channel_partner_id 上时,结果会在大约 0.2 到 0.7 秒内返回最大 channel_administrators.channel_partner_id 和 50 个结果我不会'预计它不会超过 50 秒。

我希望时间增加在最坏的情况下是线性的,但时间增加似乎不止于此。这种非线性增加让我觉得我做某事(非常?)错了,但我不知道如何找出我的查询有什么问题。谁能告诉我为什么这个查询中的时间会非线性增加?

我在 post.

底部包含了一些测试查询 运行 和它们的最新时间

编辑: 我认为这种现象的最好例子是看测试 2 和测试 3。这些例子尽可能地精简,它表明 运行 逻辑一次进行得很快,但 50 次进行得非常非常慢。

编辑 2: 我添加了更多数据,在 6.93 秒而不是 75+ 秒内获得了相同的结果。对于我的系统,我认为这是一个 acceptable 结果。我现在就写一篇这个问题的答案。

80 秒查询:

SELECT 
    info.managed_id,
    info.channel_name,
    info.registered_users, 
    info.new_users, 
    info.active_users, 
    info.coupon_opens
    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(users.id) 
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
        ) AS registered_users,
        (
            SELECT COUNT(DISTINCT users.id)
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
            AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS new_users,
        (
            SELECT COUNT(DISTINCT logins.user_id)
            FROM logins
            WHERE logins.channel_partner_id = channel_partners.id
            AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS active_users,
        (
            SELECT COUNT(coupon_trackings.id) AS coupon_view_count
            FROM coupon_trackings
            WHERE coupon_trackings.channel_partner_id = channel_partners.id
            AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS coupon_opens

    FROM channel_partners
) AS info
ON managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

ORDER BY info.channel_name

1.5 秒查询(注释掉差异):

SELECT 
    info.managed_id,
    info.channel_name,
    info.registered_users, 
    info.new_users, 
--     info.active_users, 
    info.coupon_opens
    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(users.id) 
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
        ) AS registered_users,
        (
            SELECT COUNT(DISTINCT users.id)
            FROM users
            WHERE users.channel_partner_id = channel_partners.id
            AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS new_users,
--         (
--             SELECT COUNT(DISTINCT logins.user_id)
--             FROM logins
--             WHERE logins.channel_partner_id = channel_partners.id
--             AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
--         ) AS active_users,
        (
            SELECT COUNT(coupon_trackings.id) AS coupon_view_count
            FROM coupon_trackings
            WHERE coupon_trackings.channel_partner_id = channel_partners.id
            AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS coupon_opens

    FROM channel_partners
) AS info
ON managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

ORDER BY info.channel_name

以下是我用来在结果最大的频道上测试个别时间的查询。

测试 1:0.441s - 对于单个最大通道:

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE channel_partners.id = 3255770

测试 2:0.368 秒 - 最大频道的活跃用户数:

SELECT COUNT(DISTINCT logins.user_id)
FROM logins
WHERE logins.channel_partner_id = 3255770
AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days

测试三:75.2s 只需登录信息

SELECT 
    info.managed_id,
    info.channel_name,
    info.active_users    
FROM channel_administrators

LEFT JOIN (    
    SELECT 
        channel_partners.id AS managed_id,
        channel_partners.name as channel_name,
        (
            SELECT COUNT(DISTINCT logins.user_id)
            FROM logins
            WHERE logins.channel_partner_id = channel_partners.id
            AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
        ) AS active_users

    FROM channel_partners
) AS info
ON info.managed_id = channel_administrators.channel_partner_id

WHERE channel_administrators.user_id = 54184

测试 4:6.93 秒 - 重写取得进展

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE (channel_partners.id IN (SELECT 
        channel_administrators.channel_partner_id
    FROM channel_administrators
    WHERE channel_administrators.user_id = 54184
    )
) 

编辑: 添加查询结果(删除名称)

80 秒查询:

|managed_id|registered_users|new_users|active_users|coupon_opens|
|----------|----------------|---------|------------|------------|
|14        |1146            |46       |282         |893         |
|27        |2159            |48       |206         |635         |
|15        |2039            |68       |490         |2560        |
|16        |15              |0        |1           |0           |
|20        |1391            |53       |413         |1614        |
|21        |3               |0        |0           |0           |
|43        |1051            |36       |255         |1234        |
|44        |706             |19       |85          |276         |
|46        |16              |0        |4           |8           |
|47        |68              |1        |5           |30          |
|48        |169             |6        |40          |308         |
|49        |408             |13       |118         |434         |
|52        |52              |1        |11          |54          |
|53        |378             |11       |111         |391         |
|54        |34              |1        |5           |57          |
|75        |576             |7        |59          |145         |
|3255347   |773             |12       |99          |167         |
|685131    |142             |0        |9           |91          |
|76        |22              |0        |9           |25          |
|55        |276             |5        |68          |251         |
|56        |2232            |79       |534         |1644        |
|57        |78              |0        |10          |47          |
|58        |708             |10       |109         |364         |
|59        |1274            |42       |465         |1929        |
|60        |133             |0        |37          |97          |
|3         |0               |0        |127         |257         |
|2144749   |0               |0        |4           |40          |
|61        |629             |9        |119         |363         |
|63        |857             |36       |267         |892         |
|64        |49              |1        |13          |21          |
|65        |723             |15       |281         |1152        |
|66        |77              |0        |17          |48          |
|67        |123             |10       |59          |190         |
|68        |693             |8        |191         |387         |
|70        |80              |0        |31          |58          |
|71        |214             |1        |41          |102         |
|72        |104             |2        |23          |49          |
|3255770   |3149            |86       |542         |2280        |
|3255771   |3012            |39       |526         |2056        |
|77        |180             |9        |89          |239         |
|477       |677             |5        |286         |583         |
|478       |335             |191      |235         |2226        |
|479       |162             |12       |51          |159         |
|480       |57              |0        |8           |12          |
|302       |51              |3        |17          |32          |
|303       |213             |37       |116         |598         |
|373109    |9               |3        |6           |4           |
|373110    |10              |2        |5           |0           |
|373111    |29              |9        |16          |29          |
|3255810   |0               |0        |0           |0           |

2秒查询:

|managed_id|registered_users|new_users|coupon_opens|
|----------|----------------|---------|------------|
|14        |1146            |46       |893         |
|27        |2159            |48       |635         |
|15        |2039            |68       |2560        |
|16        |15              |0        |0           |
|20        |1391            |53       |1614        |
|21        |3               |0        |0           |
|43        |1051            |36       |1234        |
|44        |706             |19       |276         |
|46        |16              |0        |8           |
|47        |68              |1        |30          |
|48        |169             |6        |308         |
|49        |408             |13       |434         |
|52        |52              |1        |54          |
|53        |378             |11       |391         |
|54        |34              |1        |57          |
|75        |576             |7        |145         |
|3255347   |773             |12       |167         |
|685131    |142             |0        |91          |
|76        |22              |0        |25          |
|55        |276             |5        |251         |
|56        |2232            |79       |1644        |
|57        |78              |0        |47          |
|58        |708             |10       |364         |
|59        |1274            |42       |1929        |
|60        |133             |0        |97          |
|3         |0               |0        |257         |
|2144749   |0               |0        |40          |
|61        |629             |9        |363         |
|63        |857             |36       |892         |
|64        |49              |1        |21          |
|65        |723             |15       |1152        |
|66        |77              |0        |48          |
|67        |123             |10       |190         |
|68        |693             |8        |387         |
|70        |80              |0        |58          |
|71        |214             |1        |102         |
|72        |104             |2        |49          |
|3255770   |3149            |86       |2280        |
|3255771   |3012            |39       |2056        |
|77        |180             |9        |239         |
|477       |677             |5        |583         |
|478       |335             |191      |2226        |
|479       |162             |12       |159         |
|480       |57              |0        |12          |
|302       |51              |3        |32          |
|303       |213             |37       |598         |
|373109    |9               |3        |4           |
|373110    |10              |2        |0           |
|373111    |29              |9        |29          |
|3255810   |0               |0        |0           |

我仍然不知道为什么这 运行 慢,但我找到了一个提高速度的解决方案,并将显示我如何到达那里的步骤。

首先,在原始问题的测试 2 和测试 3 中,您可以看到对登录 table 的单个查询非常快,但是在查询多个 channel_partner_id 时整个查询变得很慢。我怀疑这与我检查 channel_partner_id 与登录 table.

的方式有关

我重写了查询以从列表而不是 select 中获取 channel_partner_id。最终结果如下所示:

SELECT 
    channel_partners.id AS managed_id,
    channel_partners.name as channel_name,
    (
        SELECT COUNT(users.id) 
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
    ) AS registered_users,
    (
        SELECT COUNT(DISTINCT users.id)
        FROM users
        WHERE users.channel_partner_id = channel_partners.id
        AND users.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS new_users,
    (
        SELECT COUNT(DISTINCT logins.user_id)
        FROM logins
        WHERE logins.channel_partner_id = channel_partners.id
        AND logins.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS active_users,
    (
        SELECT COUNT(coupon_trackings.id) AS coupon_view_count
        FROM coupon_trackings
        WHERE coupon_trackings.channel_partner_id = channel_partners.id
        AND coupon_trackings.created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days
    ) AS coupon_opens

FROM channel_partners

WHERE (channel_partners.id IN (SELECT 
        channel_administrators.channel_partner_id
    FROM channel_administrators
    WHERE channel_administrators.user_id = 54184
    )
) 

这个查询用了 6.93 秒,这仍然很慢,但它更接近我预期的查询时间。

我无法解释为什么一种方式更快而另一种方式更慢。

每个 table 都需要这个 复合 索引,其中的列按给定顺序排列:

INDEX(channel_partner_id, created_at)

如果您有 channel_partner_id,则删除相应的索引。

将这个“覆盖”索引添加到channel_administrators

INDEX(user_id, channel_partner_id)

并删除 INDEX(user_id)(如果存在)。

通过不嵌套选择来简化查询:

    SELECT  cp.id AS managed_id,
            cp.name as channel_name,
            (   SELECT  COUNT(users.id)
                    FROM  users
                    WHERE  users.channel_partner_id = cp.id 
            ) AS registered_users,
            ((etc))
        FROM  channel_partners AS cp
        JOIN  channel_administrators AS ca
                ON cp.managed_id = ca.channel_partner_id
        WHERE  ca.user_id = 54184
        ORDER  BY  channel_name 

提示:考虑更改

created_at BETWEEN '2021-06-03' AND '2021-07-03' -- 30 days

    created_at >= '2021-06-03'
AND created_at  < '2021-06-03' + INTERVAL 30 DAY

或使用 + INTERVAL 1 MONTH(如果更合适)。

如果您仍然有性能问题,让我们看看查询是什么样的提供SHOW CREATE TABLE.