1

I'm trying to get the number of unique events on a specific date, rolling 90/30/7 days back. I've got this working on a limited number of rows with the query bellow but for large data sets I get memory errors from the aggregated string which becomes massive.

I'm looking for a more effective way of achieving the same result.

Table looks something like this:

+---+------------+-------------+
|   |     date   |     userid  |
+---+------------+-------------+
| 1 | 2013-05-14 | xxxxx       |
| 2 | 2017-03-14 | xxxxx       |
| 3 | 2018-01-24 | xxxxx       |
| 4 | 2013-03-21 | xxxxx       |
| 5 | 2014-03-19 | xxxxx       |
| 6 | 2015-09-03 | xxxxx       |
| 7 | 2014-02-06 | xxxxx       |
| 8 | 2014-10-30 | xxxxx       |
| ..| ...        | ...         |
+---+------------+-------------+

Format of the desired result:

+---+------------+---------------------------------------------+
|   |     date   | active_users_7_days | active_users_90_days  |
+---+------------+---------------------------------------------+
| 1 | 2013-05-14 | 1240                | 34339                 |
| 2 | 2017-03-14 | 4334                | 54343                 |
| 3 | 2018-01-24 | .....               | .....                 |
| 4 | 2013-03-21 | .....               | .....                 |
| 5 | 2014-03-19 | .....               | .....                 |
| 6 | 2015-09-03 | .....               | .....                 |
| 7 | 2014-02-06 | .....               | .....                 |
| 8 | 2014-10-30 | .....               | .....                 |
| ..| ...        | .....               | .....                 |
+---+------------+---------------------------------------------+

My query looks like this:

#standardSQL
    WITH
      T1 AS(
      SELECT
        date,
        STRING_AGG(DISTINCT userid) AS IDs
      FROM
        `consumer.events`
      GROUP BY
        date ),
      T2 AS(
      SELECT
        date,
        STRING_AGG(IDs) OVER(ORDER BY UNIX_DATE(date) RANGE BETWEEN 90 PRECEDING
          AND CURRENT ROW) AS IDs
      FROM
        T1 )
    SELECT
      date,
      (
      SELECT
        COUNT(DISTINCT (userid))
      FROM
        UNNEST(SPLIT(IDs)) AS userid) AS NinetyDays
    FROM
      T2
4
  • Why do you want a massive STRING_AGG(DISTINCT userid)? Apr 16, 2018 at 9:05
  • @FelipeHoffa I think I need the distinct userids grouped by date. Do you have any other more efficient ways to achieve the result?
    – Frithiof
    Apr 16, 2018 at 9:43
  • @Frithiof I think what Felipe is asking is do you need to show the actual ids or will a count of unique suffice? It's the aggregation of the strings that causing the memory error, unless you actually need to see them all then simply return a count instead.
    – Ben P
    Apr 16, 2018 at 9:52
  • @BenP Yes the aggregation of the strings are causing the error. I don't need to see the actual IDs. Maybe I'm being slow but how would I calculate the distinct IDs in a date range without aggregating them?
    – Frithiof
    Apr 16, 2018 at 10:26

2 Answers 2

5

Counting unique users requires a lot of resources, even more if you want results over a rolling window. For a scalable solution, look into approximate algorithms like HLL++:

For an exact count, this would work (but gets slower as the window gets larger):

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , COUNT(DISTINCT owner_user_id) unique_90_day_users
 , COUNT(DISTINCT IF(i<31,owner_user_id,null)) unique_30_day_users
 , COUNT(DISTINCT IF(i<8,owner_user_id,null)) unique_7_day_users
FROM (
  SELECT DATE(creation_date) date, owner_user_id
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1, 2
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
ORDER BY date_grp

enter image description here

The approximate solution produces results way faster (14s vs 366s, but then the results are approximate):

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , HLL_COUNT.MERGE(sketch) unique_90_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
FROM (
  SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
ORDER BY date_grp

enter image description here


Updated query that gives correct results - removing rows with less than 90 days (works when no dates are missing):

#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
 , HLL_COUNT.MERGE(sketch) unique_90_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
 , HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
 , COUNT(*) window_days
FROM (
  SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
  FROM `bigquery-public-data.stackoverflow.posts_questions` 
  WHERE EXTRACT(YEAR FROM creation_date)=2017
  GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
HAVING window_days=90
ORDER BY date_grp
9
  • Thank you Felipe, this was what I was looking for. Will look into HLL++ as well.
    – Frithiof
    Apr 17, 2018 at 9:11
  • The results will however be inaccurate in the first 90 days and last 90 days of the full date range. My query will be more correct since it rolls back 90 days per row. Any way to make that possible?
    – Frithiof
    Apr 17, 2018 at 11:01
  • Easy: drop the first 90 days of results, or enlarge the range by 90 days. Apr 17, 2018 at 11:26
  • Correct me if I'm wrong but if I would enlarge the range by 90 days, I would populate dates in the future which makes no sense. Would I use LIMIT and OFFSET for this?
    – Frithiof
    Apr 17, 2018 at 13:26
  • 2
    To have results up-to date, do DATE_ADD instead of DATE_SUB. It gives you the exact same results, the difference is it list the last date in the 90 day period, instead of the first day. But you get the most current 90 days regardless. Apr 18, 2018 at 13:56
1

You can aggregate the date and do the sum. What is the aggregation? Take the most recent date:

select count(*) as num_users,
       sum(case when date > datediff(current_date, interval -30 day) then 1 else 0 end) as num_users_30days,
       sum(case when date > datediff(current_date, interval -60 day) then 1 else 0 end) as num_users_60days,
       sum(case when date > datediff(current_date, interval -90 day) then 1 else 0 end) as num_users_90days
from (select user_id, max(date) as max(date)
      from `consumer.events` e
      group by user_id
     ) e;

If the most recent date for the user is in the period, then the user should be counted.

You can get this "as-of" a particular date by using a where clause in the subquery.

1
  • Thanks for that Gordon but I need the results per date for all the days in the table. I have edited my question with the format I want.
    – Frithiof
    Apr 16, 2018 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.