3

Google says BigQuery can handle billions of rows.

For my application I estimate a usage of 200,000,000 * 1000 rows. Well over a few billion.

I can partition data into 200,000,000 rows per partition but the only support for this in BigQuery seems to be different tables. (please correct me if I am wrong)

The total data size will be around 2TB.

I saw in the examples some large data sizes, but the rows were all under a billion.

Can BigQuery support the number of rows I am dealing with in a single table?

If not, can I partition it in any way besides multiple tables?

2 Answers 2

8

Below should answer your question

I run it agains one of our dataset
As you can see tables size close to 10TB with around 1.3-1.6 Billion rows

SELECT 
  ROUND(size_bytes/1024/1024/1024/1024) as TB, 
  row_count as ROWS
FROM [mydataset.__TABLES__] 
ORDER BY row_count DESC
LIMIT 10

I think the max table we dealt so far was at least up to 5-6 Billion and all worked as expected

Row   TB        ROWS     
1   10.0    1582903965   
2   11.0    1552433513   
3   10.0    1526783717   
4    9.0    1415777124   
5   10.0    1412000551   
6   10.0    1410253780   
7   11.0    1398147645   
8   11.0    1382021285   
9   11.0    1378284566   
10  11.0    1369109770   
7
  • Promising, but I am handling two orders of magnitude more rows.
    – BAR
    Oct 2, 2015 at 23:26
  • Forgot to mention - this are daily partitioned data/tables - one table for one day. For some analysis we have to query far more than just one table. It is obvious, but - BigQuery is columnar storage so you control how much you query by using only fields you really need. Oct 2, 2015 at 23:31
  • Same case with my data - partitioned by day. I could also use another index to increase partitions. How are you accomplishing the partitioning? By using different tables?
    – BAR
    Oct 2, 2015 at 23:37
  • Somthing like this - each day goes to new daily table - logname_YYYYMMDD Oct 2, 2015 at 23:38
  • BigQuery - for active queriable data Oct 2, 2015 at 23:43
3

Short answer: Yes, BigQuery will handle this just fine, even if you put all the data in a single table.

If you do want to partition your data, the only way to do it right now is to explicitly store your data in multiple tables. You might consider doing so to reduce your bill if you frequently query only a subset of your data. Many users partition their data by date and use table wildcard functions to write queries across a subset of those partitioned tables.

7
  • The huge issue with a single table or many small tables is pricing! Google will charge for all the data ($5 / TB) in the requested column even if you pulled just one row! And if your tables are too small, the minimum they will charge for any query is 10MB!
    – BAR
    Oct 4, 2015 at 17:19
  • This is clearly an optimization problem, but Ive found it has a very limited solution space, mainly focused on the quantity of data one will be querying and how well you can get the queries to be batched by 10MB. It would be worthwhile for a company spending 10k+ per month on data storage to optimize for.
    – BAR
    Oct 4, 2015 at 17:22
  • @BAR to store your 2TB of data that is just $40 per month. Why do you say it's 10k+ per month? Put your data in bigquery for a few days and see what's the pricing, then you will be surprised how low is.
    – Pentium10
    Oct 5, 2015 at 7:05
  • @Pentium10 no, i said neither of those things. I am talking query costs not storage. And i am not spending 10k, but it would be worth some optimization for a company that is..
    – BAR
    Oct 5, 2015 at 16:14
  • 1
    @BAR, what's your use case? BigQuery's query engine and pricing structure are optimized for analytic queries that scan a significant chunk of data per query. If you're doing point lookups where your goal is to pull out a single record, you may have better luck with a system like Google Datastore, which is designed for transactional workloads instead. Oct 5, 2015 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.