I am processing data using Google DataFlow is there any way to write the results to cloudsql ?
1 Answer
There is not currently a built-in way of doing this, but you can use the Sink API to write to arbitrary external systems.
Edit: The Sink API is likely overkill for writing to cloudsql; you can just directly make insert calls from a DoFn.
-
Thanks danielm. Is it a good practice to write to cloudsql from dataflow ? Nov 11, 2016 at 21:45
-
1Sure, you can write to any external data storage system you need from Dataflow. Depending on your usecase, writing to BigQuery may also work, and Dataflow supports that out of the box.– danielmNov 11, 2016 at 23:22
-
1I wouldn't say that SinkAPI is overkill, since it allows you to write to CloudSQL in a transaction - either everything is saved or nothing. Sep 19, 2017 at 12:46