BigQuery：如何避免 "Resources exceeded during query execution." 错误

Question

我想知道如何避免 "resources exceeded during execution" 错误。关于此的大多数其他问题都涉及 JOIN EACH 或 GROUP EACH BY，但我已经没有使用这些。如果我在日期或 ABS(HASH(userId)) 上包含 WHERE 子句，则查询有效，但我希望整个数据集可用，然后我将在 Tableau 中进一步过滤它。

如果我删除 t4 查询有效，但我想要最后一列，并且我想要从 event_parameters 字段中创建更多列以供以后查询。

作业 ID 是 rhi-localytics-db:job_6MaesvuMK6mP6irmAnrcM9R3cx8 如果有帮助，谢谢。

SELECT
    t1.userId as userId,
    t1.event_time AS event_time,
    t1.Diamond_Balance as Diamond_Balance,
    t2.Diamond_Change as Diamond_Change,
    t3.Gold_Balance as Gold_Balance,
    t4.Gold_Change as Gold_Change
FROM (
    SELECT
        userId,
        event_time,
        INTEGER(event_parameters.Value) AS Diamond_Balance,
    FROM
        FLATTEN([game_data], event_parameters)
    WHERE
        event_name LIKE 'Currency'
        AND event_parameters.Name = 'Diamond_Balance'
        -- and date(event_time) > '2015-09-11'
        -- AND ABS(HASH(userId) % 5)  = 0
    GROUP BY
        userId,
        event_time,
        Diamond_Balance ) AS t1
INNER JOIN (
    SELECT
        userId,
        event_time,
        INTEGER(event_parameters.Value) AS Diamond_Change,
    FROM
        FLATTEN([game_data], event_parameters)
    WHERE
        event_name LIKE 'Currency'
        AND event_parameters.Name = 'Diamond_Change'
        AND INTEGER(event_parameters.Value ) < 14000
        AND INTEGER(event_parameters.Value ) > -14000
        -- and date(event_time) > '2015-09-11'
        -- AND ABS(HASH(userId) % 5)  = 0

    GROUP BY
        userId,
        event_time,
        Diamond_Change ) AS t2
ON
    t1.userId = t2.userId
    AND t1.event_time = t2.event_time
INNER JOIN (
    SELECT
        userId,
        event_time,
        event_parameters.Value AS Gold_Balance,
    FROM
        FLATTEN([game_data], event_parameters)
    WHERE
        event_name LIKE 'Currency'
        AND event_parameters.Name = 'Gold_Balance'
        -- and date(event_time) > '2015-09-11'
        -- AND ABS(HASH(userId) % 5)  = 0

    GROUP BY
        userId,
        event_time,
        Gold_Balance ) AS t3
ON
    t1.userId = t3.userId
    AND t1.event_time = t3.event_time
INNER JOIN (
    SELECT
        userId,
        event_time,
        INTEGER(event_parameters.Value) AS Gold_Change,
    FROM
        FLATTEN([game_data], event_parameters)
    WHERE
        event_name LIKE 'Currency'
        AND event_parameters.Name = 'Gold_Change'
        -- and date(event_time) > '2015-09-11'
        -- AND ABS(HASH(userId) % 5)  = 0
    GROUP BY
        userId,
        event_time,
        Gold_Change ) AS t4
ON
    t1.userId = t4.userId
    AND t1.event_time = t4.event_time

Answer 1

我认为您应该能够在一个简单的 "scan" 中完成所有逻辑。
完全没有加入！
像下面这样的东西。只是想法 - 但有一些机会按原样工作:)

SELECT
    userId,
    event_time,
    MAX(CASE WHEN event_parameters.Name = 'Diamond_Balance' 
            THEN INTEGER(event_parameters.Value) END) AS Diamond_Balance,
    MAX(CASE WHEN event_parameters.Name = 'Diamond_Change' AND INTEGER(event_parameters.Value ) BETWEEN -14000 AND 14000 
            THEN INTEGER(event_parameters.Value)) END AS Diamond_Change,
    MAX(CASE WHEN event_parameters.Name = 'Gold_Balance' 
            THEN INTEGER(event_parameters.Value) END) AS Gold_Balance,
    MAX(CASE WHEN event_parameters.Name = 'Gold_Change' 
            THEN INTEGER(event_parameters.Value) END) AS Gold_Change
FROM
    FLATTEN([game_data], event_parameters)
WHERE
    event_name LIKE 'Currency'
GROUP BY
    userId,
    event_time

Answer 2

有关超出资源的一般建议可在此处找到：

请注意，添加 EACH 通常是资源超出错误的解决方案，而不是其原因。（尽管在某些情况下它可以反过来工作！）

此外，EACH 在 GROUP BY 上不再有意义，并且很快就会在 JOIN 上变得无关紧要。

BigQuery：如何避免 "Resources exceeded during query execution." 错误

BigQuery: How to Avoid "Resources exceeded during query execution." error

sql

google-bigquery