在 bigquery 中对行和中位数进行计算——这可能吗？

Question

我的问题是获取一组原始的感官数据，在我使用它之前需要进行一些处理。将数据加载到客户端并进行处理非常慢，因此寻找将此逻辑卸载到 bigquery 的可能性。

假设我有一组传感器的一些常量。它们可以改变，但是当我想做查询时我有它们

A: 1, B: 2, C: 3, D: 2, E: 1, F: 2

传感器已连接，我知道哪些传感器相互连接。含义如下。

A: BC
D: EF

这是 table 每个传感器每个时间戳的测量值。想象一下数千行。

TS    A  |  B  |  C  |  D  |  E  |  F  
01    10 |  20 |  10 |  10 |  15 | 10
02    11 |  10 |  20 |  20 |  10 | 10
03    12 |  20 |  10 |  10 |  12 | 11
04    13 |  10 |  10 |  20 |  15 | 15
05    11 |  20 |  10 |  15 |  14 | 14
06    10 |  20 |  10 |  10 |  15 | 12

我想查询 ts 01 到 ts 06（实际上它可能又是 1000 行）。我不希望它 return 这个原始数据，但让它做一些计算：

首先，对于每一行，我需要减去常量，所以第 01 行看起来像：

01    9 |  18 |  17 |  8 |  14 | 8

那么，BC需要减去A，EF需要减去D：

01    9 |   9 |   8 |  8 |   6 | 0

最后一步，当我拥有所有行时，我想要 return 行，其中每个传感器都具有该传感器的后续 X 行的中值。所以

    TS    A  |  B  | 
    01    10 |  1  | 
    02    11 |  2  |  
    03    12 |  2  |  
    04    13 |  1  |  
    05    11 |  2  | 
    06    10 |  3  |  
    07    10 |  4  | 
    08    11 |  2  |  
    09    12 |  2  |  
    10    13 |  10 |  
    11    11 |  20 | 
    12    10 |  20 |

returns（X 为 4）

    TS    A  |  B  | 
   //first 3 needed for median for 4th value
    04    11.5 |  etc  |   //median 10, 11, 12, 13
    05    11.5 |  etc  |   //median 11, 12, 13, 11
    06    11.5 |  etc  |   //median 12, 13, 11, 10
    07    etc |  etc | 
    08    etc |  etc |  
    09    etc |  etc |  
    10    etc |  etc |  
    11    etc |  etc | 
    12    etc |  etc |

将数据获取到我的服务器并进行计算非常慢，我真的很想知道我们是否可以在 bigQuery 中获取这些数据量，因此我能够使用我自己选择的设置快速计算出一组数据!

我在 Node.js 中这样做...但是在 BigQuery SQL 中..我迷路了。

Answer 1

以下适用于 BigQuery 标准 SQL

如果您要查找 AVG 值 - 这将是 "simple" 如下

#standardSQL
WITH constants AS (
  SELECT 1 val_a, 2 val_b, 3 val_c, 2 val_d, 1 val_e, 2 val_f
), temp AS (
  SELECT ts,
    a - val_a AS a, 
    b - val_b - a + val_a AS b,
    c - val_c - a + val_a AS c,
    d - val_d AS d,
    e - val_e - d + val_d  AS e,
    f - val_f - d + val_d AS f
  FROM `project.dataset.measurements`, constants
)
SELECT ts, 
  AVG(a) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) a,  
  AVG(b) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) b,  
  AVG(c) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) c,  
  AVG(d) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) d,  
  AVG(e) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) e,  
  AVG(f) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) f  
FROM temp

对于 MEDIAN，您需要添加一些额外的东西——如下例所示

#standardSQL
WITH constants AS (
  SELECT 1 val_a, 2 val_b, 3 val_c, 2 val_d, 1 val_e, 2 val_f
), temp AS (
  SELECT ts,
    a - val_a AS a, 
    b - val_b - a + val_a AS b,
    c - val_c - a + val_a AS c,
    d - val_d AS d,
    e - val_e - d + val_d  AS e,
    f - val_f - d + val_d AS f
  FROM `project.dataset.measurements`, constants
)
SELECT ts,
  (SELECT PERCENTILE_CONT(a, 0.5) OVER() FROM UNNEST(a) a LIMIT 1) a,
  (SELECT PERCENTILE_CONT(b, 0.5) OVER() FROM UNNEST(b) b LIMIT 1) b,
  (SELECT PERCENTILE_CONT(c, 0.5) OVER() FROM UNNEST(c) c LIMIT 1) c,
  (SELECT PERCENTILE_CONT(d, 0.5) OVER() FROM UNNEST(d) d LIMIT 1) d,
  (SELECT PERCENTILE_CONT(e, 0.5) OVER() FROM UNNEST(e) e LIMIT 1) e,
  (SELECT PERCENTILE_CONT(f, 0.5) OVER() FROM UNNEST(f) f LIMIT 1) f
FROM (
  SELECT ts, 
    ARRAY_AGG(a) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) a,  
    ARRAY_AGG(b) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) b,  
    ARRAY_AGG(c) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) c,  
    ARRAY_AGG(d) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) d,  
    ARRAY_AGG(e) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) e,  
    ARRAY_AGG(f) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) f  
  FROM temp
)

您可以使用您问题中的示例数据来测试和使用上面的示例，如下例所示

#standardSQL
WITH `project.dataset.measurements` AS (
  SELECT 01 ts, 10 a, 20 b, 20 c, 10 d, 15 e, 10 f UNION ALL
  SELECT 02, 11, 10, 20, 20, 10, 10 UNION ALL
  SELECT 03, 12, 20, 10, 10, 12, 11 UNION ALL
  SELECT 04, 13, 10, 10, 20, 15, 15 UNION ALL
  SELECT 05, 11, 20, 10, 15, 14, 14 UNION ALL
  SELECT 06, 10, 20, 10, 10, 15, 12 
), constants AS (
  SELECT 1 val_a, 2 val_b, 3 val_c, 2 val_d, 1 val_e, 2 val_f
), temp AS (
  SELECT ts,
    a - val_a AS a, 
    b - val_b - a + val_a AS b,
    c - val_c - a + val_a AS c,
    d - val_d AS d,
    e - val_e - d + val_d  AS e,
    f - val_f - d + val_d AS f
  FROM `project.dataset.measurements`, constants
)
SELECT ts,
  (SELECT PERCENTILE_CONT(a, 0.5) OVER() FROM UNNEST(a) a LIMIT 1) a,
  (SELECT PERCENTILE_CONT(b, 0.5) OVER() FROM UNNEST(b) b LIMIT 1) b,
  (SELECT PERCENTILE_CONT(c, 0.5) OVER() FROM UNNEST(c) c LIMIT 1) c,
  (SELECT PERCENTILE_CONT(d, 0.5) OVER() FROM UNNEST(d) d LIMIT 1) d,
  (SELECT PERCENTILE_CONT(e, 0.5) OVER() FROM UNNEST(e) e LIMIT 1) e,
  (SELECT PERCENTILE_CONT(f, 0.5) OVER() FROM UNNEST(f) f LIMIT 1) f
FROM (
  SELECT ts, 
    ARRAY_AGG(a) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) a,  
    ARRAY_AGG(b) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) b,  
    ARRAY_AGG(c) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) c,  
    ARRAY_AGG(d) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) d,  
    ARRAY_AGG(e) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) e,  
    ARRAY_AGG(f) OVER(ORDER BY ts ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING) f  
  FROM temp
)
-- ORDER BY ts

结果

Row ts  a       b       c       d       e       f    
1   1   null    null    null    null    null    null     
2   2   9.0     9.0     8.0     8.0     6.0     0.0  
3   3   9.5     3.5     7.5     13.0    -1.5    -5.0     
4   4   10.0    7.0     7.0     8.0     3.0     0.0  
5   5   10.5    2.5     1.5     13.0    -0.5    -2.5     
6   6   10.5    2.5     -3.5    15.5    -2.0    -3.0

在 bigquery 中对行和中位数进行计算——这可能吗？

Do calculations in bigquery within row and median - is this possible?

calculated-columns

median

google-bigquery