MySQL：无法 select 来自特定分区的记录？

Question

我正在使用 MySQL 5.6。我创建了一个包含 366 个分区的 table 来按天保存数据，这意味着一年中我们最多有 366 天，所以我在 table 上创建了 366 个分区。散列分区由一个整数列管理，该列为每条记录存储 1 到 366。

Report_Summary 表:

CREATE TABLE `Report_Summary` (
  `PartitionsID` int(4) unsigned NOT NULL,
  `ReportTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `Amount` int(10) NOT NULL,
  UNIQUE KEY `UNIQUE` (`PartitionsID`,`ReportTime`),
  KEY `PartitionsID` (`PartitionsID`),
  KEY `ReportTime` (`ReportTime`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=COMPRESSED
/*!50100 PARTITION BY HASH (PartitionsID)
PARTITIONS 366 */

我当前查询：

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= '2014-12-26 00:00:00' AND RS.ReportTime <= '2014-12-30 23:59:59' AND 
      RS.PartitionsID BETWEEN DAYOFYEAR('2014-12-26 00:00:00') AND DAYOFYEAR('2014-12-30 23:59:59')
GROUP BY ReportDate;

以上查询运行良好，并使用分区 p360 到 p364 来获取数据。现在的问题是当我将 fromDate 传递给 '2014-12-26' 并将 toDate 传递给 '2015-01-01' 然后上面的查询将不起作用。因为“2015-01-01”的日期是 1 所以我的条件失败了。

现在我尝试在 IN 运算符中传递值，然后它在查询下面的数据库检查中完美运行：

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= '2014-12-26 00:00:00' AND RS.ReportTime <= '2015-01-01 23:59:59' AND 
      RS.PartitionsID IN (360,361,362,363,364,365,1)
GROUP BY ReportDate;

为了生成上述场景，我创建了一个函数并传递了两个日期并生成了一个逗号分隔的 ID 字符串

SELECT GenerateRange('2014-12-26 00:00:00', '2015-01-01 23:59:59');

返回我的数据为：

'360,361,362,363,364,365,366,1'

并且我尝试在我的查询中使用该函数，所以我更改了我的查询如下：

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= '2014-12-26 00:00:00' AND RS.ReportTime <= '2015-01-01 23:59:59' AND 
      FIND_IN_SET(RS.PartitionsID, GenerateRange('2014-12-26 00:00:00', '2015-01-01 00:00:00'))
GROUP BY ReportDate;

然后我使用 EXPLAIN PARTITION SELECT... 检查了上述查询的执行计划。我发现我的条件行不通了。它使用所有分区来获取数据。我只想使用那些日期的特定分区。必须只检查这些 360,361,362,363,364,365,366,1 分区意味着 p360 到 p366 和 p1.

为什么我的查询不起作用？这不是实现这个的正确方法然后我想要解决方案我怎样才能实现这个？

我从编码中知道我可以实现它，但我必须编写一个查询来实现它。

谢谢...

Answer 1

我能想到几个选项。

创建 case 涵盖多年搜索条件的报表。
创建一个 CalendarDays table 并使用它为您的 in 子句获取 DayOfYear 的不同列表。
选项 1 的变体，但使用 union 分别搜索每个范围

选项 1： 使用 case 语句。它不漂亮，但似乎有效。在这种情况下，如果查询跨越非闰年的年份，则此选项可以搜索一个额外的分区 366。此外，我不确定优化器是否会喜欢 RS.ParitionsID 过滤器中的 OR，但您可以尝试一下。

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= @startDate AND RS.ReportTime <= @endDate
    AND 
    (
    RS.PartitionsID BETWEEN 
        CASE 
            WHEN
                --more than one year, search all days 
                year(@endDate) - year(@startDate) > 1
                --one full year difference 
                OR year(@endDate) - year(@startDate) = 1 
                    AND DAYOFYEAR(@startDate) <= DAYOFYEAR(@endDate)
            THEN 1
            ELSE DAYOFYEAR(@startDate)
        END
        and 
        CASE
            WHEN 
                --query spans the end of a year
                year(@endDate) - year(@startDate) >= 1
            THEN 366
            ELSE DAYOFYEAR(@endDate)
        END
    --Additional query to search less than portion of next year
    OR RS.PartitionsID <=
        CASE
            WHEN year(@endDate) - year(@startDate) > 1
                OR DAYOFYEAR(@startDate) > DAYOFYEAR(@endDate)
            THEN DAYOFYEAR(@endDate)
            ELSE NULL
        END
    )
GROUP BY ReportDate;

选项 2： 使用 CalendarDays table。这个选项要干净得多。缺点是你需要创建一个新的 CalendarDays table 如果你没有

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= @startDate AND RS.ReportTime <= @endDate
    AND RS.PartitionsID IN
    (
        SELECT DISTINCT DAYOFYEAR(c.calDate) 
        FROM dbo.calendarDays c
        WHERE c.calDate >= @startDate and c.calDate <= @endDate
    )

编辑：选项 3： 选项 1 的变体，但使用 Union All 分别搜索每个范围。这里的想法是，由于语句中没有 OR，优化器将能够应用分区修剪。注意：我通常不在 MySQL 工作，所以我的语法可能有点不对劲，但大意是有的。

DECLARE @startDate datetime, @endDate datetime;
DECLARE @rangeOneStart datetime, @rangeOneEnd datetime, @rangeTwoStart datetime, @rangeTwoEnd datetime;

SELECT @rangeOneStart := 
        CASE 
            WHEN
                --more than one year, search all days 
                year(@endDate) - year(@startDate) > 1
                --one full year difference 
                OR year(@endDate) - year(@startDate) = 1 
                    AND DAYOFYEAR(@startDate) <= DAYOFYEAR(@endDate)
            THEN 1
            ELSE DAYOFYEAR(@startDate)
        END
    , @rangeOneEnd := 
        CASE
            WHEN 
                --query spans the end of a year
                year(@endDate) - year(@startDate) >= 1
            THEN 366
            ELSE DAYOFYEAR(@endDate)
        END 
    , @rangeTwoStart := 1
    , @rangeTwoEnd := 
        CASE
            WHEN year(@endDate) - year(@startDate) > 1
                OR DAYOFYEAR(@startDate) > DAYOFYEAR(@endDate)
            THEN DAYOFYEAR(@endDate)
            ELSE NULL
        END
;

SELECT t.ReportDate, sum(t.Amount) as Total
FROM 
(
    SELECT DATE(RS.ReportTime) AS ReportDate, RS.Amount
    FROM Report_Summary RS
    WHERE RS.PartitionsID BETWEEN @rangeOneStart AND @rangeOneEnd
        AND RS.ReportTime >= @startDate AND RS.ReportTime <= @endDate

    UNION ALL

    SELECT DATE(RS.ReportTime) AS ReportDate, RS.Amount
    FROM Report_Summary RS
    WHERE RS.PartitionsID BETWEEN @rangeTwoStart AND @rangeTwoEnd
        AND @rangeTwoEnd IS NOT NULL
        AND RS.ReportTime >= @startDate AND RS.ReportTime <= @endDate
) t
GROUP BY ReportDate;

Answer 2

要开始解决这个问题，您需要一个子查询，给定一个日期范围，return 一个包含该范围内所有 DAYOFYEAR() 值的结果集。

让我们来解决这个问题。对于初学者，我们需要一个查询，该查询可以 return 从 0 到至少 366 的所有整数的序列。这是该查询。它返回一列 seq 值 0-624。

SELECT A.N + 5*(B.N + 5*(C.N + 5*(D.N))) AS seq
  FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 
                      UNION SELECT 3 UNION SELECT 4) AS A
  JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                      UNION SELECT 3 UNION SELECT 4) AS B
  JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                      UNION SELECT 3 UNION SELECT 4) AS C
  JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                      UNION SELECT 3 UNION SELECT 4) AS D

（这是生成 5**4 数字的所有组合的简单交叉连接技巧。）

接下来，我们需要使用它来生成 DAYOFYEAR() 值列表。为了示例，让我们使用您的开始日期和结束日期。此查询生成一个结果集，其中包含一串整数，显示该日期范围内一年中的天数。

SELECT DISTINCT DAYOFYEAR(first_day + INTERVAL seq DAY) doy
  FROM (SELECT DATE('2014-12-26 00:00:00') AS first_day,
               DATE('2015-01-01 23:59:59') AS last_day
       ) params
  JOIN (
         SELECT A.N + 5*(B.N + 5*(C.N + 5*(D.N))) AS seq
           FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 
                               UNION SELECT 3 UNION SELECT 4) AS A
           JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                               UNION SELECT 3 UNION SELECT 4) AS B
           JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                               UNION SELECT 3 UNION SELECT 4) AS C
           JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                               UNION SELECT 3 UNION SELECT 4) AS D
       ) seq ON seq.seq <= TIMESTAMPDIFF(DAY,first_day,last_day)
 ORDER BY 1

我认为您可以说服自己这个粗糙的小查询在大约一年半（625 天）或更短时间内的任何合理天数范围内都能正常工作。如果你使用更长的时间跨度，你可能会弄乱闰年。

最后，您可以在 PartitionsID IN () 子句中使用此查询。看起来像这样。

SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
  FROM Report_Summary RS
 WHERE RS.ReportTime >= '2014-12-26 00:00:00'
   AND RS.ReportTime <= '2015-01-01 23:59:59'
   AND RS.PartitionsID 
     IN (
         SELECT DISTINCT DAYOFYEAR(first_day + INTERVAL seq DAY) doy
           FROM (SELECT DATE('2014-12-26 00:00:00') AS first_day,
                        DATE('2015-01-01 23:59:59') AS last_day
                ) params
           JOIN (
                  SELECT A.N + 5*(B.N + 5*(C.N + 5*(D.N))) AS seq
                    FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 
                                        UNION SELECT 3 UNION SELECT 4) AS A
                    JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                                        UNION SELECT 3 UNION SELECT 4) AS B
                    JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                                        UNION SELECT 3 UNION SELECT 4) AS C
                    JOIN (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2
                                        UNION SELECT 3 UNION SELECT 4) AS D
                ) seq ON seq.seq <= TIMESTAMPDIFF(DAY,first_day,last_day)
          ORDER BY 1
         ) 
GROUP BY ReportDate;

这应该适合你。

如果您使用的是 MariaDB 10+，则有 built in sequence tables 命名为 seq_0_to_624。

这里有一篇关于这个主题的文章：

http://www.plumislandmedia.net/mysql/filling-missing-data-sequences-cardinal-integers/

Answer 3

我得到了解决方案，因为我更改了在 table 中存储 PartitionsId 列的逻辑。最初我将 DayOfYear(reportTime) 列存储在 PartitionsId 列中。现在我通过存储 TO_DAYS(reportTime) 并存储到 PartitionsId 列中改变了逻辑。

所以我的table结构如下：

CREATE TABLE `Report_Summary` (
  `PartitionsID` int(10) unsigned NOT NULL,
  `ReportTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `Amount` int(10) NOT NULL,
  UNIQUE KEY `UNIQUE` (`PartitionsID`,`ReportTime`),
  KEY `PartitionsID` (`PartitionsID`),
  KEY `ReportTime` (`ReportTime`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=COMPRESSED
/*!50100 PARTITION BY HASH (PartitionsID)
PARTITIONS 366 */

INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735928','2014-12-26 11:46:12','100');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735929','2014-12-27 11:46:23','50');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735930','2014-12-28 11:46:37','44');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735931','2014-12-29 11:46:49','15');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735932','2014-12-30 11:46:59','56');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735933','2014-12-31 11:47:22','68');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735934','2015-01-01 11:47:35','76');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735935','2015-01-02 11:47:43','88');
INSERT INTO `Report_Summary` (`PartitionsID`, `ReportTime`, `Amount`) VALUES('735936','2015-01-03 11:47:59','77');

勾选 SQL FIDDLE DEMO:

我的查询是：

EXPLAIN PARTITIONS 
SELECT DATE(RS.ReportTime) AS ReportDate, SUM(RS.Amount) AS Total
FROM Report_Summary RS
WHERE RS.ReportTime >= '2014-12-26 00:00:00' AND RS.ReportTime <= '2015-01-01 23:59:59' AND 
      RS.PartitionsID BETWEEN TO_DAYS('2014-12-26 00:00:00') AND TO_DAYS('2015-01-01 23:59:59')
GROUP BY ReportDate;

上面的查询扫描了我需要的特定分区，它也使用了正确的索引。因此，在更改 PartitionsId 列的逻辑后，我找到了正确的解决方案。

感谢大家的回复，非常感谢大家的宝贵时间...

Answer 4

根据您的 SELECT，您真正需要的是一种名为 "Summary Tables" 的数据仓库技术。这样，您可以每天（或每小时或其他时间）汇总数据并将小计存储在更小的 table 中。然后 "report" 查看 table 并汇总小计。这通常比原始数据的强力扫描快 10 倍。更多详情：http://mysql.rjweb.org/doc.php/datawarehouse .

这样做消除了在原始数据 ("Fact table") 或摘要 table.

中进行分区的需要

但是，如果您需要清除旧数据，那么 PARTITIONing 可以派上用场，因为 DROP PARTITION。为此，您将使用 BY RANGE(TO_DAYS(...))，而不是 BY HASH。

MySQL：无法 select 来自特定分区的记录？

MySQL: Unable to select the records from specific partitions?

mysql

sql

select

partitioning

find-in-set