筛选所需的分区字段 n unnest array of date

Question

我需要查询我的 table 由必需的 transactionDate 字段分区。

我只想在多个日期过滤我的数据以避免恢复大量数据，但我遇到了这个错误：Query error: Cannot query over table t without a filter over column(s) 'transactionDate' that can be used for partition elimination

我的代码：

DECLARE dates ARRAY < DATE >;

SET
dates = (
    SELECT
        ARRAY_AGG(DISTINCT(DATE(transactionDate)))
    FROM
        t
    WHERE
        transactionDate >= "1900-01-01"
        AND analyticsUpdateDate BETWEEN "2022-05-01"
        AND "2022-05-10"
);

SELECT * FROM t WHERE transactionDate IN UNNEST(dates)

如何在不查询整个 table 的情况下处理此错误？

感谢您的帮助。

Answer 1

当 dates 为空时，transactionDate IN UNNEST(dates) 条件将为 false，您的查询将 return 出错：

Cannot query over table 'testset.t' without a filter over column(s) 'transactionDate' that can be used for partition elimination

DECLARE dates ARRAY <DATE>;

-- Sample table
CREATE SCHEMA IF NOT EXISTS testset;
CREATE OR REPLACE TABLE testset.t PARTITION BY (transactionDate) OPTIONS (require_partition_filter = true) AS
SELECT n, p AS transactionDate, p AS analyticsUpdateDate
  FROM UNNEST(GENERATE_ARRAY(0, 9)) n WITH OFFSET 
 CROSS JOIN UNNEST(GENERATE_DATE_ARRAY('2022-04-01', '2022-04-10')) p WITH OFFSET USING(OFFSET)
;

-- Your Query will retrun an error because `dates` is empty.
SET dates = (
    SELECT
        ARRAY_AGG(DISTINCT(DATE(transactionDate)))
    FROM
        testset.t
    WHERE
        transactionDate >= "1900-01-01"
        AND analyticsUpdateDate BETWEEN "2022-05-01"
        AND "2022-05-10"
);

SELECT * FROM testset.t WHERE transactionDate IN UNNEST(dates);

解决此问题的简单方法是在分区列上添加 IS NOT NULL 条件，如下所示。

SELECT * FROM testset.t 
 WHERE transactionDate IS NOT NULL AND transactionDate IN UNNEST(dates);

筛选所需的分区字段 n unnest array of date

Filter required partition field n unnest array of date

sql

google-bigquery

google-cloud-platform