15

The google documentation only talks about daily partitions. But is there anything in the model that hinders one from stuffing partitions in a table with other time period (e.g., hour or week)?

Are there any limits or drawbacks from having partitions in a "small" table?

1
  • 1
    You have a low rate. Important on SO, you have to mark accepted answers by using the tick on the left of the posted answer, below the voting. This will increase your rate. See how this works by visinting this link: meta.stackoverflow.com/questions/5234/…
    – Pentium10
    Aug 8, 2018 at 20:52

3 Answers 3

9

Currently only DAY partitioned tables is supported. Hourly or monthly is not supported. There are several feature request for new functionality but there is no timeline for implementation. You can comment and add your use case on the tickets as well to spread the word.

https://cloud.google.com/bigquery/docs/partitioned-tables

related feature requests:

Update * august 2018

Introduction to Clustered Tables - You have now a way to partition by day, and then further cluster your table by any others column(s) such as hour/minute.

Clustering can improve the performance of certain types of queries such as queries that use filter clauses and queries that aggregate data. When data is written to a clustered table by a query job or a load job, BigQuery sorts the data using the values in the clustering columns. These values are used to organize the data into multiple blocks in BigQuery storage. When you submit a query containing a clause that filters data based on the clustering columns, BigQuery uses the sorted blocks to eliminate scans of unnecessary data.

1
  • 2
    Clustering isn't really a solution for some cases. Example: You create your table based on a CSV file which doesn't contain any hour column or even timestamp. Furthermore if this table creation is using the "load" API, it is not even possible to add a timestamp column after-the-fact. So even with clustering this doesn't help "partition further" timewise, instead it just "groups" by some other unrelated types of columns... Oct 12, 2018 at 6:01
4

Yes, now Big Query supports hourly partitioning on Ingestion time.

Here is the documentation

4

Yes.
Big Query now supports hourly partitioning based on any TIMESTAMP field, and not just on Ingestion time, by using TIMESTAMP_TRUNC:

 CREATE TABLE
   mydataset.newtable (transaction_id INT64,
     transaction_ts TIMESTAMP)
 PARTITION BY
   TIMESTAMP_TRUNC(transaction_ts, HOUR)
 OPTIONS
   ( partition_expiration_days=3,
     description="a table partitioned by transaction_ts" )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.