5

I was trying to load data from Google's json export, but it looks like it's not valid JSON (ECMA-404),(RFC 7159),(RFC 4627). Here is what I'm expecting for json newline:

[{},{},{}]

But here is what it's giving:

{}{}{}

Here's an example output from clicking the "Download as JSON" button on a four-row query result:

{"c0":"001U0000016lf5jIAA","c1":"Tim Burton's Corpse Bride","c2":"a0KU000000OkQ8IMAV","c3":"Luxembourg","c4":"German","c5":"Sub & Audio","c21":null,"c22":"2025542.0"}
{"c0":"001U0000016lf5jIAA","c1":"Tim Burton's Corpse Bride","c2":"a0KU000000OkQ8IMAV","c3":"Luxembourg","c4":"German","c5":"Sub & Audio","c21":null,"c22":"2025542.0"}
{"c0":"001U0000016lf5jIAA","c1":"Tim Burton's Corpse Bride","c2":"a0KU000000OjUuEMAV","c3":"Luxembourg","c4":"French - Parisian","c5":"Sub & Audio","c21":null,"c22":"2025542.0"}
{"c0":"001U0000016lf5jIAA","c1":"Tim Burton's Corpse Bride","c2":"a0KU000000OkQ8IMAV","c3":"Luxembourg","c4":"German","c5":"Sub & Audio","c21":null,"c22":"2025542.0"}

Is there a reason why BigQuery is using this export format for json? Are there other Google services or something that are dependent on this format, or why would it be pushing a non-standard json format? (Maybe I'm just misunderstanding json line format). Note, this is from the web-UI, not the API, which gives valid json.

5
  • when you say - what it's giving - what exactly you mean!? Mar 6, 2018 at 23:45
  • @MikhailBerlyant see updated response.
    – David542
    Mar 6, 2018 at 23:50
  • question is crystal clear now :o) Mar 6, 2018 at 23:50
  • 1
    Using the new UI (sign up through the alpha form) I get the same format with the "JSON" result tab as with "Save As" and selecting "Download as JSON", if that helps. Mar 6, 2018 at 23:53
  • @ElliottBrossard thank you -- just submitted, thanks.
    – David542
    Mar 7, 2018 at 0:04

1 Answer 1

5

BigQuery reads and outputs new-line delimited JSON - this because traditional JSON doesn't adapt well to the needs of big data.

See:

The output of "Download as JSON" shown in the question is compatible with the JSON input that BigQuery can read.

Note that the web UI also offers to look at the results of a query as JSON - and those results are formatted as a traditional JSON object. I'm not sure what was the design decision to have this incompatible output here - but results in that form won't be able to be imported back to BigQuery.

So in general, this format is incompatible with BigQuery:

enter image description here

While this is compatible with BigQuery:

enter image description here

Why is this less traditional JSON format the best choice in the big data world? Encapsulating a trillion rows within [...] defines a single object with a trillion rows - which is hard to parse and handle. New line delimited JSON solves this problem, with each row being an independent object.

1
  • 1
    thanks, this is very clear and explains the difference well.
    – David542
    Mar 8, 2018 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.