4

We are using Bigquery streaming inserts with Dataflow using the predefined Dataflow job template.

I ran into some peculiarities when using this with nullable and repeated fields.

For instance, with the schema

name   STRING, NULLABLE

Attempting to do the insert {name: null}

fails with the error:

generic::invalid_argument: This field is not a record.","location":"name","message":"This field is not a record."

This is not such a big deal since it's easy enough to simply drop null fields, and similarly for empty arrays.

However, now if our schema is:

name   STRING, REPEATED

and we want to insert ["a", "b", null, "c"] we get a similar error referencing the third element.

1 Answer 1

2

To provide a row with a null value for a NULLABLE field, simply omit the field from the row that you are inserting. For your second example, a REPEATED field (or an ARRAY in SQL terms) cannot have a null element. To model an array of NULLABLE STRING, you can use a REPEATED RECORD that contains a STRING field named value, for instance, or equivalently an ARRAY<STRUCT<value STRING>> in SQL terms.

2
  • Thanks! For posterity, I think part of my confusion here was due to the way loading a datastore export into bigquery creates the schema for structured properties. Repeated structured properties with singular fields are loaded into bigquery as singular records with repeated fields (I have no clue how this works for repeated structured properties with repeated fields...) I was able to change the schema to what Elliott suggested and everything worked.
    – Viknesh
    Commented Oct 19, 2017 at 20:43
  • @Viknesh how did you resolve your issue? Did you go with the ARRAY<STRUCT<value STRING>> approach listed here?
    – user780092
    Commented Feb 17, 2022 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.