1

We have over 100m rows in big query of analytics data. Each record is an event attached to an id.

A simplification:

ID  EventId  Timestamp

Is it possible to flatten this to one table holding rows like:

ID timestamp-period event1 event2 event3 event4

Where the event columns hold the counts of the number of events for that id in that time period?

So far, i've managed to do it on small data sets with 2 queries. One to create rows that hold counts for an individual event id and another to flatten these in to one row after. The reason I haven't yet been able to do this accross the whole data set is that bigquery runs out of resources - not entirely sure why.

These two queries look something like this:

SELECT 
VideoId,
date_1,
IF(EventId = 1, INTEGER(count), 0) AS user_play,
IF(EventId = 2, INTEGER(count), 0) AS auto_play,
IF(EventId = 3, INTEGER(count), 0) AS pause,
IF(EventId = 4, INTEGER(count), 0) AS replay,
IF(EventId = 5, INTEGER(count), 0) AS stop,
IF(EventId = 6, INTEGER(count), 0) AS seek,
IF(EventId = 7, INTEGER(count), 0) AS resume,
IF(EventId = 11, INTEGER(count), 0) AS progress_25,
IF(EventId = 12, INTEGER(count), 0) AS progress_50,
IF(EventId = 13, INTEGER(count), 0) AS progress_75,
IF(EventId = 14, INTEGER(count), 0) AS progress_90,
IF(EventId = 15, INTEGER(count), 0) AS data_loaded,
IF(EventId = 16, INTEGER(count), 0) AS playback_complete,
IF(EventId = 30, INTEGER(count), 0) AS object_click,
IF(EventId = 31, INTEGER(count), 0) AS object_rollover,
IF(EventId = 32, INTEGER(count), 0) AS object_clickthrough,
IF(EventId = 33, INTEGER(count), 0) AS object_shown,
IF(EventId = 34, INTEGER(count), 0) AS object_close,
IF(EventId = 40, INTEGER(count), 0) AS logo_clickthrough,
IF(EventId = 41, INTEGER(count), 0) AS endframe_clickthrough,
IF(EventId = 42, INTEGER(count), 0) AS startframe_clickthrough,
IF(EventId = 61, INTEGER(count), 0) AS share_facebook,
IF(EventId = 62, INTEGER(count), 0) AS share_twitter,
IF(EventId = 63, INTEGER(count), 0) AS open_social_panel,
IF(EventId = 70, INTEGER(count), 0) AS embed_code_requested,
IF(EventId = 80, INTEGER(count), 0) AS player_impression,
IF(EventId = 81, INTEGER(count), 0) AS player_loaded,
IF(EventId = 90, INTEGER(count), 0) AS html5_impression,
IF(EventId = 91, INTEGER(count), 0) AS html5_load,
IF(EventId = 95, INTEGER(count), 0) AS fallback_impression,
IF(EventId = 96, INTEGER(count), 0) AS fallback_load,
IF(EventId = 152, INTEGER(count), 0) AS object_impression,
IF(EventId = 200, INTEGER(count), 0) AS ping,
IF(EventId = 250, INTEGER(count), 0) AS facebook_clickthrough,
IF(EventId = 251, INTEGER(count), 0) AS twitter_clickthrough,
IF(EventId = 252, INTEGER(count), 0) AS other_clickthrough,
IF(EventId = 253, INTEGER(count), 0) AS qr_clickthrough,
IF(EventId = 254, INTEGER(count), 0) AS banner_clickthrough,
IF(EventId = 280, INTEGER(count), 0) AS banner_impression,
IF(EventId = 281, INTEGER(count), 0) AS banner_loaded,
IF(EventId = 282, INTEGER(count), 0) AS banner_data_loaded,
IF(EventId = 284, INTEGER(count), 0) AS banner_forward,
IF(EventId = 285, INTEGER(count), 0) AS banner_back,
IF(EventId = 300, INTEGER(count), 0) AS mobile_preview_loaded,
IF(EventId = 301, INTEGER(count), 0) AS mobile_preview_clickthrough,
IF(EventId = 302, INTEGER(count), 0) AS mobile_preview_clickthrough_back,
IF(EventId = 310, INTEGER(count), 0) AS product_search_click,
IF(EventId = 311, INTEGER(count), 0) AS promo_code_click,
IF(EventId = 320, INTEGER(count), 0) AS player_share_facebook,
IF(EventId = 321, INTEGER(count), 0) AS player_share_twitter,
IF(EventId = 322, INTEGER(count), 0) AS player_share_googleplus,
IF(EventId = 323, INTEGER(count), 0) AS player_share_email,
IF(EventId = 324, INTEGER(count), 0) AS player_share_embed,
IF(EventId = 401, INTEGER(count), 0) AS youtube_error_2,
IF(EventId = 402, INTEGER(count), 0) AS youtube_error_100,
IF(EventId = 403, INTEGER(count), 0) AS youtube_error_101,
FROM
(
SELECT 
  VideoId, EventId, count(*) as count, Date(timestamp) as date_1  
FROM [data.data_1]
GROUP EACH BY VideoId, EventId, date_1
)
ORDER BY data_loaded DESC;

Then just a group by on id and timestamp creates the full aggregated table.

Am I doing this the right way, and do I just need to do it on a small partition of the dataset or is there a better way to aggregate like this that will use bigquery in a more efficient way?

Thanks in advance, Mat

1 Answer 1

1

My guess is that you're running out of resources because of the ORDER BY at the end. Everything else should be able to be done in parallel. Also note that if you remove the order by, you will be able to use the 'allow large results' flag and write out a large table of the results (if the results are > 128MB).

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.