5

I'm trying to get all rows associated with a particular entity ID over the last week. Each row has a timestamp. I'd like to group those entities by day, by extracting the date from the datetime timestamp, but when I try, I get

Error: (L6:28): expression STRFTIME_UTC_USEC([data_timestamp], '%Y-%m-%d') in GROUP BY is invalid

from bigquery. It appears as though something is wrong with the DATE() function.

Here is the full query I am running:

SELECT Count(*) FROM [myproj:mydataset.mytable] 
WHERE 
    bool_property=False 
AND 
    entity_id=5667423172689920 
AND
    DATEDIFF(CURRENT_DATE(), data_timestamp) <= 7
GROUP BY DATE(data_timestamp)

Looking at the data, it would appear that the quality, here called 'data_timestamp' is in fact a proper timestamp; I really don't understand why the DATE() function would be failing.

Any help or tips? Thanks!

2 Answers 2

10

There are two problems with your query

  1. in BigQuery Legacy SQL you cannot use expression in GROUP BY statement, rather just field (btw, this limitation does not exist in BigQuery Standard SQL)
  2. When you do GROUP BY - you cannot just simply select out all fields, rather you need use some aggregation functions like COUNT or SUM etc (for those fields which are not part of GROUP VY)

So your query can look something like below:

SELECT DATE(data_timestamp) AS dt, COUNT(1) AS cnt 
FROM [myproj:mydataset.mytable] 
WHERE bool_property=FALSE 
AND entity_id=5667423172689920 
AND DATEDIFF(CURRENT_DATE(), data_timestamp) <= 7
GROUP BY dt
5
  • well, it appears I am using standard sql, since i've tested other queries using 'GROUP BY', and they seemed to work fine. You're right about the aggregate function thing though. I changed my question to fit. My real issue is not with GROUP BY, but with the DATE function, which seems to illogically not work.
    – bgenchel
    Commented Aug 29, 2016 at 3:56
  • based on syntax yo use - you are using legacy sql. have you tried to run query in my answer? the problem is not in date function but in fact that you used it in group by statement in legacy sql Commented Aug 29, 2016 at 3:58
  • Yeah that totally worked! Though, i'm not sure I understand why. I suppose your theory/hypothesis is that the date function was not working out because it was in GROUP BY? Mind explaining a bit more how this query works?
    – bgenchel
    Commented Aug 29, 2016 at 4:02
  • as i explained already - in bigquery legacy sql you cannot use expression in group by statement, rather yo need to precalculate this expression in select statement DATE(data_timestamp) AS dt and then use that new "field" (dt) in group by. this is usual approach Commented Aug 29, 2016 at 4:05
  • ah I see now, it's not that GROUP BY doesn't work, it's expressions inside. Awesome. Thanks so much!
    – bgenchel
    Commented Aug 29, 2016 at 5:14
1

Unfortunatelly legacy SQL doesn't allow GROUP BY expressions, only by column names. But with Standard SQL you can do it:

SELECT * FROM myproj.mydataset.mytable
WHERE 
    bool_property=False AND 
    entity_id=5667423172689920 AND
    DATE_DIFF(CURRENT_DATE(), EXTRACT(DATE FROM data_timestamp), DAY) <= 7
GROUP BY EXTRACT(DATE FROM data_timestamp)
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.