47

I am storing data in unixtimestamp on google big query. However, when the user will ask for a report, she will need the filtering and grouping of data by her local timezone.

The data is stored in GMT. The user may wish to see the data in EST. The report may ask the data to be grouped by date.

I don't see the timezone conversion function here:

Does anyone know how I can do this in bigquery? i.e. how do i group by after converting the timestamp to a different timezone?

7 Answers 7

64

Standard SQL in BigQuery has built-in functions:

DATE(timestamp_expression, timezone)
TIME(timestamp, timezone)
DATETIME(timestamp_expression, timezone)

Example:

SELECT 
   original,
   DATETIME(original, "America/Los_Angeles") as adjusted
FROM sometable;

+---------------------+---------------------+
| original            | adjusted            |
+---------------------+---------------------+
| 2008-12-25 05:30:00 | 2008-12-24 21:30:00 |
+---------------------+---------------------+

You can use standard IANA timezone names or offsets.

35

As of September 2016 BigQuery has adopted standard SQL and you can now just use the "DATE(timestamp, timezone)" function to offset for a timezone. You can reference their docs here:

BigQuery DATE docs

5
  • 1
    It is true that this functionality is new, but it does not solve the problem. If you have a timestamp, you typically want a new timestamp. This function takes in a timestamp and outputs a date. Commented Feb 16, 2017 at 10:35
  • 1
    @DennisJaheruddin A timestamp is an absolute point in time (microseconds from the Unix epoch for BigQuery). There is no "new" timestamp to get. If you want that point in time represented in a specific time zone then you have to convert that absolute time into a logical calendar time, usually represented by date. This is exactly what this function does. Commented Apr 10, 2017 at 17:53
  • @ManiGandham I don't understand your response, what I am talking about is that this command changes something like 1-jan-2001 07:00:00 (don't mind the format) into 1 jan 2001, so it is only the date without the time. Commented Apr 11, 2017 at 11:10
  • 3
    @DennisJaheruddin Did you read the documentation? You can use DATETIME(timestamp, timezone) instead if you need a DateTime result. cloud.google.com/bigquery/docs/reference/standard-sql/… Commented Apr 11, 2017 at 14:35
  • In that case I either missed it or the docs have been updated since I checked in February. Either way, it is nice that you posted this answer. Commented Apr 12, 2017 at 8:27
9

To those that stumble here:

How to convert a timestamp to another timezone?

Given that TIMESTAMP values, once constructed, are stored as UTC, and that TIMESTAMP does not have a constructor (TIMESTAMP, STRING), you can convert a timestamp to another time zone by transforming it first to a DATETIME and then constructing the new TIMESTAMP from the DATETIME in the new timezone:

SELECT TIMESTAMP(DATETIME(timestamp_field, '{timezone}'))

Example:

SELECT
    input_tz,
    input,
    'America/Montreal' AS output_tz,
    TIMESTAMP(DATETIME(input,'America/Montreal')) AS output
FROM (
    SELECT 'US/Pacific' AS input_tz, TIMESTAMP(DATETIME(DATE(2021, 1, 1), TIME(16, 0, 0)), 'US/Pacific') AS input
    UNION ALL
    SELECT 'UTC' AS input_tz, TIMESTAMP(DATETIME(DATE(2021, 1, 1), TIME(16, 0, 0)), 'UTC') AS input
    UNION ALL
    SELECT 'Europe/Berlin' AS input_tz, TIMESTAMP(DATETIME(DATE(2021, 1, 1), TIME(16, 0, 0)), 'Europe/Berlin') AS input
) t

results in:

Row input_tz input output_tz output
1 US/Pacific 2021-01-02 00:00:00 UTC America/Montreal 2021-01-01 19:00:00 UTC
2 UTC 2021-01-01 16:00:00 UTC America/Montreal 2021-01-01 11:00:00 UTC
3 Europe/Berlin 2021-01-01 15:00:00 UTC America/Montreal 2021-01-0110:00:00 UTC

How to strip time zone info from a DATETIME value?

DATETIME in BigQuery are time zone naive, such that they do not contain timezone info. This being said, if you have business knowledge that allows you to know the timezone of a DATETIME, you can strip that timezone offset by converting it to a TIMESTAMP with the known timezone:

SELECT TIMESTAMP(datetime_value, '{timezone}')

Given that the TIMESTAMP stores the value in UTC, you can then re-convert to DATETIME if that's your preferred method of storage, but now you'll know that your DATETIME is in UTC :)

Hopefully this can be helpful! :)

2
  • This is not correct. You need to move the parenthesis on the timestamp to the end, otherwise it adds 4 hours, not subtract. TIMESTAMP(DATETIME(CURRENT_TIMESTAMP(), 'America/Montreal')) Commented Sep 25, 2021 at 18:29
  • @MikeDeluca You are absolutely right, correcting this right away! Commented Oct 5, 2021 at 22:54
4

Your premise is right. If you group like this, then users who want EST or EDT will get incorrect date grouping:

GROUP BY UTC_USEC_TO_DAY(ts_field)

But as long as you figure out the offset that your user wants, you can still do the full calculation on the server. For example, if EST is 5 hours behind UTC then query like this:

GROUP BY UTC_USEC_TO_DAY(ts_field - (5*60*60*1000*1000000) )

Just parameterize the "5" to be the offset in hours, and you're all set. Here's a sample based on one of the sample data sets:

SELECT
  COUNT(*) as the_count,
  UTC_USEC_TO_DAY(timestamp * 1000000 - (5*60*60*1000*1000000) ) as the_day
FROM
  [publicdata:samples.wikipedia]
WHERE
  comment CONTAINS 'disaster'
  and timestamp >= 1104537600
GROUP BY
  the_day
ORDER BY
  the_day

You can remove the offset to see how some edits move to different days.

3
  • 1
    Hi, thanks for the feedback. this would def work for straightforward conversions. However, the daylight savings time throw a wrench in the problem as you can't just use a static value.
    – akshah123
    Commented Sep 21, 2012 at 13:46
  • Agreed: you need your app (or your users!) to figure out the offset they want. That's not really an ideal solution. You'll definitely miss out on edge cases like what are the exact boundaries of "today" on the day Daylight Saving Time starts or ends. But it would cover many needs.
    – mdahlman
    Commented Sep 21, 2012 at 17:58
  • we are following the same approach.
    – pankajmi
    Commented Dec 17, 2014 at 10:45
4

To convert any TimeZone DateTime string to UTC, one could use PARSE_TIMESTAMP using the supported TIMESTAMP Formats in BigQuery.

For example to convert IST (Indian Standard Time) string to UTC, use the following:

SAFE.PARSE_TIMESTAMP("%a %b %d %T IST %Y", timeStamp_vendor, "Asia/Kolkata")

Here PARSE_TIMESTAMP parses the IST string to a UTC TIMESTAMP (not string). Adding SAFE as prefix takes care of errors/nulls etc.

To convert this to a readable string format in BigQuery, use FORMAT_TIMESTAMP as follows:

FORMAT_TIMESTAMP("%d-%b-%Y %T %Z", SAFE.PARSE_TIMESTAMP("%a %b %d %T IST %Y", timeStamp_vendor, "Asia/Kolkata"))

This example would take an IST string of the format Fri May 12 09:45:12 IST 2019 and convert it to 12-May-2019 04:15:12 UTC.

Replace IST with the required TimeZone and Asia/Kolkata with relevant Timezone name to achieve the conversion for your timezone

3

2016 update: Look answers below, BigQuery now provides timestamp and timezone methods.


You are right - BigQuery doesn't provide any timestamp conversion methods.

In this case, I suggest that you run your GROUP BY based on dimensions of the GMT/UTC timestamp field, and then convert and display the result in the local timezone in your code.

7
  • Hi Michael, Thanks for the response. the problem is that suppose I need data grouped by date. Now, the data needs to be grouped by the date in local timezone. i.e. if its 1 am today in GMT, its actually yesterday in EST. I would have to group data by each timestamp value and return it. However, this will require a lot of network IO and a very slow performance on overall report. Is there any hacks I can do as a work around?
    – akshah123
    Commented Sep 18, 2012 at 18:35
  • 1
    Why not have a column that records the user's local timezone with each record? Then you can quickly iterate through the resulting rows and provide the proper display date. Commented Sep 18, 2012 at 18:42
  • we did consider this option but the problem is that multiple users may request to see the same data. So, it is not practical so store the value in every possible conversion with the record. However, for now, we only have users in EST so we can choose this route. This would be a problem once we have users outside of the region.
    – akshah123
    Commented Sep 19, 2012 at 13:29
  • By the way, is there somewhere I can suggest this feature to be added in big query?
    – akshah123
    Commented Sep 19, 2012 at 13:31
  • 1
    UPDATE: we have started a new Google BigQuery public issue tracker to help surface feature requests and bug reports: code.google.com/p/google-bigquery Commented Jan 18, 2013 at 11:06
-2

For me TIMESTAMP_SUB and TIMESTAMP_ADD functions did the job. When needed to convert timestamp from UTC to PST I used:

TIMESTAMP_SUB(`timestamp`, INTERVAL 8 HOUR)
1
  • Only use this as a last resort. PST only exists during some period of the year, so if you use this for PDT, for instance, you will get the incorrect date. Moreover, if you have a time zone value like "America/Los_Angeles", this approach wont help at all. Check other answers Commented Jul 29, 2021 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.