从一组范围计算并发性

Question

我有一组包含开始时间戳和持续时间的行。我想使用重叠或并发执行各种摘要。

例如：每日并发峰值，并发峰值分组在另一列。

示例数据：

timestamp,duration
2016-01-01 12:00:00,300
2016-01-01 12:01:00,300
2016-01-01 12:06:00,300

我想知道该期间的峰值是 12:01:00-12:05:00 并发 2 个。

关于如何使用 BigQuery 或不那么令人兴奋的 Map/Reduce 工作来实现此目标的任何想法？

Answer 1

对于每分钟的解决方案，会话长度最长为 255 分钟：

SELECT session_minute, COUNT(*) c
FROM (
  SELECT start, DATE_ADD(start, i, 'MINUTE') session_minute FROM (
    SELECT * FROM (
      SELECT TIMESTAMP("2015-04-30 10:14") start, 7 minutes
    ),(
      SELECT TIMESTAMP("2015-04-30 10:15") start, 12 minutes
    ),(
      SELECT TIMESTAMP("2015-04-30 10:15") start, 12 minutes
    ),(
      SELECT TIMESTAMP("2015-04-30 10:18") start, 12 minutes
    ),(
      SELECT TIMESTAMP("2015-04-30 10:23") start, 3 minutes
    ) 
  ) a
  CROSS JOIN [fh-bigquery:public_dump.numbers_255] b
  WHERE a.minutes>b.i
)
GROUP BY 1
ORDER BY 1

Answer 2

STEP 1 - First you need find all periods (start and end) with respective concurrent entries

SELECT ts AS start, LEAD(ts) OVER(ORDER BY ts) AS finish, 
       SUM(entry) OVER(ORDER BY ts) AS concurrent_entries
FROM (
  SELECT ts, SUM(entry)AS entry 
  FROM 
    (SELECT ts, 1 AS entry FROM yourTable),
    (SELECT DATE_ADD(ts, duration, 'second') AS ts, -1 AS entry FROM yourTable)
  GROUP BY ts
  HAVING entry != 0
)
ORDER BY ts

假设输入如下

(SELECT TIMESTAMP('2016-01-01 12:00:00') AS ts, 300 AS duration),
(SELECT TIMESTAMP('2016-01-01 12:01:00') AS ts, 300 AS duration),
(SELECT TIMESTAMP('2016-01-01 12:06:00') AS ts, 300 AS duration),
(SELECT TIMESTAMP('2016-01-01 12:07:00') AS ts, 300 AS duration),
(SELECT TIMESTAMP('2016-01-01 12:10:00') AS ts, 300 AS duration),
(SELECT TIMESTAMP('2016-01-01 12:11:00') AS ts, 300 AS duration)

上述查询的输出看起来像这样：

start                       finish                      concurrent_entries   
2016-01-01 12:00:00 UTC     2016-01-01 12:01:00 UTC     1    
2016-01-01 12:01:00 UTC     2016-01-01 12:05:00 UTC     2    
2016-01-01 12:05:00 UTC     2016-01-01 12:07:00 UTC     1    
2016-01-01 12:07:00 UTC     2016-01-01 12:10:00 UTC     2    
2016-01-01 12:10:00 UTC     2016-01-01 12:12:00 UTC     3    
2016-01-01 12:12:00 UTC     2016-01-01 12:15:00 UTC     2    
2016-01-01 12:15:00 UTC     2016-01-01 12:16:00 UTC     1    
2016-01-01 12:16:00 UTC     null                        0

您可能仍想稍微完善以上查询 - 但主要是它满足您的需求

STEP 2 - now you can do any stats off of above result

例如整个周期的峰值：

SELECT 
  start, finish, concurrent_entries, RANK() OVER(ORDER BY concurrent_entries DESC) AS peak
FROM (
  SELECT ts AS start, LEAD(ts) OVER(ORDER BY ts) AS finish, 
         SUM(entry) OVER(ORDER BY ts) AS concurrent_entries
  FROM (
    SELECT ts, SUM(entry)AS entry FROM 
      (SELECT ts, 1 AS entry FROM yourTable),
      (SELECT DATE_ADD(ts, duration, 'second') AS ts, -1 AS entry FROM yourTable)
    GROUP BY ts
    HAVING entry != 0
  )
)
ORDER BY peak

从一组范围计算并发性

Calculating concurrency from a set of ranges

sql

google-app-engine

google-bigquery