15

I would like to trigger a Cloud Function when new data has been imported into a BigQuery table. Ideally, I would like to extract all the rows (one column is ISIN) that have been inserted.

Would this be possible and how?

9
  • Right now this is not possible, but stay tuned as there is something coming up soon.
    – Pentium10
    Feb 20, 2019 at 20:00
  • In the mean time, would you suggest that the best solution for the moment is to have a cron job to check if the data is there and then trigger a background function using pubsub or http?
    – WJA
    Feb 20, 2019 at 20:04
  • Create a job, and I would either list most recent jobs, and would push the jobid for further processing, or directly query the table for new rows.
    – Pentium10
    Feb 20, 2019 at 20:26
  • 1
    @Pentium10 news?
    – franco_b
    Jun 6, 2020 at 6:31
  • 2
    The “something coming soon” ... is that EventArc ( cloud.google.com/blog/products/serverless/…)?
    – jamiet
    Feb 13, 2021 at 17:49

1 Answer 1

16

Update June 2022: Cloud Functions 2nd Generation supports Eventarc

Using Eventarc it's possible to listen to a long list of cloud events and then trigger a set of consumers including Cloud Functions.

The combination of serviceName = bigquery.googleapis.com and methodName = jobservice.jobcompleted with additional custom filters can be used to create the Eventrac sink, from which Cloud Functions can be triggered. You can read more here.

This is still not a native BigQuery <> Cloud Function integration, because it uses a 3rd cloud service. Therefore it's not much different than the solution suggested before June 2022 of creating a sink with Cloud Logging. Yet, this is the better practice these days of implementing cross-service triggers on GCP.


Old answer: Unfortunately BigQuery has no triggers as a native feature as of May 2022 :/ However there is a nice workaround you can implement! :)

As you probably know, almost every operation in GCP is logged to Cloud Logging (previously StackDriver Logging). In this article they demonstrate how to trigger a Cloud Function whenever a BigQuery load job finishes. The idea is simple:

  1. Create a filter for BigQuery load jobs in Cloud Logging to catch when load jobs terminate.
  2. Create a sink from Cloud Logging to Pub/Sub topic.
  3. Set your Cloud Function as a consumer on this topic and implement your logic

Google even published this concept as an article.

3
  • Agreed with this, already tried it before and it works great
    – WJA
    Jan 3, 2021 at 0:41
  • Now Cloud Function 2nd gen, supports trigger based on BigQuery events
    – Ambesh
    Jun 20, 2022 at 21:24
  • 1
    Is there any way to make the sink send the table's name/ID to PubSub using this method? Jun 27, 2022 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.