We are using Bigquery streaming inserts with Dataflow using the predefined Dataflow job template.
I ran into some peculiarities when using this with nullable and repeated fields.
For instance, with the schema
name STRING, NULLABLE
Attempting to do the insert {name: null}
fails with the error:
generic::invalid_argument: This field is not a record.","location":"name","message":"This field is not a record."
This is not such a big deal since it's easy enough to simply drop null fields, and similarly for empty arrays.
However, now if our schema is:
name STRING, REPEATED
and we want to insert ["a", "b", null, "c"]
we get a similar error referencing the third element.