Google Bigquery 中的嵌套字段使用 OFFSET 而不是 UNNEST

Question

向 GBQ 大师提问。

这是两个目的相同的查询

first

SELECT
  fullVisitorId AS userid,
  CONCAT(fullVisitorId, visitStartTime) AS session,
  visitStartTime + (hits[
  OFFSET(0)].time / 1000) AS eventtime,
  date,
  trafficSource.campaign,
  trafficSource.source,
  trafficSource.medium,
  trafficSource.adContent,
  trafficSource.adwordsClickInfo.campaignId,
  geoNetwork.region,
  geoNetwork.city,
  trafficSource.keyword,
  totals.visits AS visits,
  device.deviceCategory AS deviceType,
  hits[OFFSET(0)].eventInfo.eventAction,
  hits[OFFSET(0)].TRANSACTION.transactionId,
  hits[OFFSET(0)].TRANSACTION.transactionRevenue,
  SUBSTR(channelGrouping,0,3) AS newchannelGrouping
FROM
  `some_site.ga_sessions_*`
WHERE
  ARRAY_LENGTH(hits) > 0
  AND _table_suffix BETWEEN '20200201'
  AND '20200201'
  AND fullVisitorId IN (
  SELECT
    DISTINCT(fullVisitorId)
  FROM
    `some_site.ga_sessions_*`,
    UNNEST(hits) AS hits
  WHERE
    _table_suffix BETWEEN '20200201'
    AND '20200201'
    AND (hits.TRANSACTION.transactionId != 'None')
)

second

SELECT
  fullVisitorId AS userid,
  CONCAT(fullVisitorId, visitStartTime) AS session,
  visitStartTime + (hits.time / 1000) AS eventtime,
  date,
  trafficSource.campaign,
  trafficSource.source,
  trafficSource.medium,
  trafficSource.adContent,
  trafficSource.adwordsClickInfo.campaignId,
  geoNetwork.region,
  geoNetwork.city,
  trafficSource.keyword,
  totals.visits AS visits,
  device.deviceCategory AS deviceType,
  hits.eventInfo.eventAction,
  hits.TRANSACTION.transactionId,
  hits.TRANSACTION.transactionRevenue,
  SUBSTR(channelGrouping,0,3) AS newchannelGrouping
FROM
  `some_site.ga_sessions_*`, UNNEST(hits) hits
WHERE
  _table_suffix BETWEEN '20200201' AND '20200201'
  AND fullVisitorId IN (
    SELECT
      DISTINCT(fullVisitorId)
    FROM
      `some_site.ga_sessions_*`,
      UNNEST(hits) AS hits
    WHERE
      _table_suffix BETWEEN '20200201'
      AND '20200201'
      AND (hits.TRANSACTION.transactionId != 'None')
  )

第一个使用 OFFSET 从嵌套字段中提取数据。根据执行详细信息报告，查询需要大约 1.5 MB 的改组。

第二个查询使用 UNNEST 来访问嵌套数据。打乱后的字节数约为 (!) 75 MB

两种情况下处理的数据量相同。

现在，问题是：

这是否意味着根据 this article 优化插槽之间的通信，我应该使用 OFFSET 而不是 UNNEST 来获取存储在嵌套字段中的数据？

谢谢！

Answer 1

让我们考虑以下使用 BigQuery public 数据集的示例。

UNNEST - returns 6 个结果：

WITH t AS (SELECT * FROM `bigquery-public-data.google_analytics_sample.ga_sessions_20170801` WHERE visitId = 1501571504 )
SELECT h FROM t, UNNEST(hits) h

OFFSET - returns 1 个结果：

WITH t AS (SELECT * FROM `bigquery-public-data.google_analytics_sample.ga_sessions_20170801` WHERE visitId = 1501571504 )
SELECT hits[OFFSET(0)] FROM t

两个查询都引用了 GA public table 中的同一条记录。他们表明，使用带有 UNNEST 的连接将为数组中的每个元素带来一行，而使用 OFFSET(0) 只会为数组的第一个元素带来一行。

造成high data shuffling差异的原因是UNNEST执行了JOIN操作，需要以特定的方式组织数据。 OFFSET 方法仅采用数组的第一个元素。

Google Bigquery 中的嵌套字段使用 OFFSET 而不是 UNNEST

Using OFFSET instead of UNNEST for nested fields in Google Bigquery

offset

google-bigquery

unnest