2

I am trying to join 2 tables, each with 57,191 rows. BigQ is looking for a larger table on the inner/left and smaller on the right. When I run it with Table B on the left, it errors as 'The large table A must appear first'. When I switch the query and put Table A in the From clause, it errors as 'The large table B must appear first'. So when I do as it instructs, it does not fix it but suggests my first (incorrect) attempt, unless I am somehow botching it.

It is a bit ironic that if the 2 tables are the same size it decides one is larger based on, presumably, that one is not smaller than the other. I am trying to find a solution which does not include me adding a meaningless row to 1 of the tables and then trying to delete it after the join works, (since BigQ is not loading my single row csv file right now, I am sure it's due to my error.)

The Google SQL syntax join rule seems to be

"join_type Bigquery supports INNER (the default) and LEFT OUTER joins. table_2 This is the second table in the join, which must be small, and will be joined to the table appearing in the FROM clause. Note that this can be either a table name or another SELECT clause, in which case you must provide an alias. join_condition_1, ..., join_condition_N, ... The set of join conditions, which must be a collection of equality conditions, all of which must be met for a row to be included in the result. (That is, we only support connecting these conditions with AND.) "

The actual SQL I am running is

SELECT lt.activeprosperloans,[fieldsredacted], ...
FROM prosperloans1.listings2 AS lt
JOIN prosperloans1.zjoinedperfloans as ln
ON lt.key = listingkey;

and the actual error reads: Error: Large table prosperloans1.zjoinedperfloans must appear as the leftmost table in a join query

Thanks Shawn

2 Answers 2

3

Note that now you can use JOIN EACH, effectively solving the problem.

Your query would become:

SELECT lt.activeprosperloans,[fieldsredacted], ...
FROM prosperloans1.listings2 AS lt
JOIN prosperloans1.zjoinedperfloans as ln
ON lt.key = listingkey;

From the docs:

When possible, use JOIN without the EACH modifier for best performance. Use JOIN EACH when table sizes are too large for JOIN.

https://developers.google.com/bigquery/docs/query-reference#joins

1

Since this question was answered, BigQuery added JOIN EACH, which is a way to join two large tables. See Fh's answer for instructions on how to use JOIN EACH.

The rest of this response is for historical purposes: A large table (for the purpose of join) is anything over 7 MB. In order to do a join, the entire small table is sent to every node in the cluster, so we place a pretty significant limit on it. It may be that despite both being the same number of rows, the one table is larger than 7 MB while the other is smaller.

One way to reduce the size of one of the tables is to apply filters and column filters in a query and save the result as another temporary table, then apply the JOIN to the temporary table. E.g. if you have 10 columns in a table that spans a month worth of data but you only need 3 columns for the join query and the last day's data, you can first just select the three columns and the recent data, and give the result a name. Then you can do the join against that table.

1
  • No worries. Since it's not really 'Big Data' I have the luxury of bringing it into Excel or Google Docs sort and merging it so will do that for now. Jul 31, 2012 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.