1

I know there's a dump of reddit comments and stories in BigQuery - as collected by Jason Baumgartner of pushshift.io.

How can I query this dataset to get a list of flairs for a subreddit?

This is the base query I have:

SELECT link_flair_text 
FROM `fh-bigquery.reddit_posts.2019_08` 
WHERE subreddit  = 'AmItheAsshole'
2

Let's build this visualization:

enter image description here

A query like this will do - this one gets the count of posts on /r/AmITheAsshole, ordered by most typical judgement:

SELECT link_flair_text, COUNT(*) c 
FROM `fh-bigquery.reddit_posts.2019_08` 
WHERE subreddit  = 'AmItheAsshole'
GROUP BY 1
ORDER BY 2 DESC 
LIMIT 1000

enter image description here

To query several months of data, you can use * on the FROM:

SELECT link_flair_text, COUNT(*) c 
FROM `fh-bigquery.reddit_posts.2019_*` 
WHERE subreddit  = 'AmItheAsshole'
GROUP BY 1
ORDER BY 2 DESC 
LIMIT 1000

enter image description here

And to see the evolution of percentages per month:

SELECT link_flair_text, ARRAY_AGG(STRUCT(month, ROUND(100*c/total_month,1) AS perc, c) ORDER BY month) AS arr
FROM (
  SELECT *, SUM(c) OVER(PARTITION BY month) total_month
  FROM (
    SELECT link_flair_text, DATE(TIMESTAMP_TRUNC(TIMESTAMP_SECONDS(created_utc), MONTH)) month, COUNT(*) c
    FROM `fh-bigquery.reddit_posts.2019_*` 
    WHERE subreddit  = 'AmItheAsshole'
    AND link_flair_text IS NOT NULL
    GROUP BY 1,2 
    HAVING c > 100
  )
)
GROUP BY 1
ORDER BY SUM(c) DESC

enter image description here

And the SQL for the viz shown above:

CREATE OR REPLACE TABLE `temp.assholes`
AS
SELECT *, c/total_month AS perc
FROM (
  SELECT *, SUM(c) OVER(PARTITION BY month) total_month, SUM(c) OVER(PARTITION BY link_flair_text) total_flair
  FROM (
    SELECT LOWER(link_flair_text) link_flair_text, DATE(TIMESTAMP_TRUNC(TIMESTAMP_SECONDS(created_utc), MONTH)) month, COUNT(*) c
    FROM `fh-bigquery.reddit_posts.201*` 
    WHERE subreddit = 'AmItheAsshole'
    AND link_flair_text IS NOT NULL
    AND _table_suffix >= '8_03'
    GROUP BY 1,2 
  )
)
WHERE c/total_month > 0.01
AND total_flair > 1000
ORDER BY month

How to bring data from before 2016?

Almost the same query, but it incorporates a different table that has all the historic content from before 2016:

WITH data AS (
  SELECT * FROM `fh-bigquery.reddit_posts.201*` WHERE _table_suffix >= '5_12'
  UNION ALL  
  SELECT * FROM `fh-bigquery.reddit_posts.full_corpus_201512`  
)

SELECT *, c/total_month AS perc
FROM (
  SELECT *, SUM(c) OVER(PARTITION BY month) total_month, SUM(c) OVER(PARTITION BY link_flair_text) total_flair
  FROM (
    SELECT LOWER(link_flair_text) link_flair_text, DATE(TIMESTAMP_TRUNC(TIMESTAMP_SECONDS(created_utc), MONTH)) month, COUNT(*) c
    FROM data
    WHERE subreddit  = 'AmItheAsshole'
    AND link_flair_text IS NOT NULL
    GROUP BY 1,2 
  )
)
WHERE c/total_month > 0.01
AND total_flair > 1000
ORDER BY month

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.