0

I have 10 requests per second of data I want to save that looks like the entry below. I need to save this data after a CloudRun function completes. (My infrastructure is on google-cloud-platform). The data will be used as a data set for machine learning.

{ 
  "text": "1k characters", 
  "text2": "1k characters", 
  "metadata1": "enum (100 vals)", 
  "metadata2": "number value" 
}

I planned to save this as a non-awaited function to google-cloud-storage either in one folder or in folders based on the metadata1 enum. Is either better than the other?

Is this the appropriate route to take?

I think pubsub is overkill as suggested in this SO answer.

5
  • Yes it is the right route to take since both Pub/Sub and Dataflow offers scalability, availability, and latency if ever you decide to ramp up the request per second in the future. You can read more about this in cloud.google.com/pubsub/architecture and cloud.google.com/dataflow/docs/guides/deploying-a-pipeline
    – Ricco D
    Sep 24, 2021 at 6:37
  • 1
    Do you want to write several messages in a file? with witch latency (or at which frequency) do you need to write the files? Sep 24, 2021 at 8:22
  • @guillaumeblaquiere the latency for writing files doesn't matter, the frequency is by the number of requests I receive which in this case is 10/s Sep 24, 2021 at 14:40
  • In what way were you thinking Pub/Sub+DF may be overkill - the engineering effort, scale, cost etc? Is the latency of completing the Cloud Run function call an important consideration? Could you clarify what you meant by "I planned to save this as a non-awaited function", do you mean have a separate Cloud Function that just writes to Cloud Storage? Sep 24, 2021 at 21:55
  • @SamarthSingal great question, engineering effort and cost. The scalability thing is great, but I don't anticipate needing more than 10 r/s, actually if anything it could go down. And I added a function to the handler that right before it sends the data to the client, it saves the JSON data to cloud storage (so yes, a separate function that just writes to cloud storage). Sep 25, 2021 at 17:11

2 Answers 2

2

I can propose you 2 patterns, but in both case you need to store the messages:

  • Either use PubSub to stack the messages. Then, use Dataflow to read pubsub and to sink to Cloud Storage. Or use a on demand service (Cloud Run for exemple) to pull your PubSub subscription and write a file with all the message read (You can trigger your Cloud Run with Cloud Scheduler, every hour for example)
  • Or store the message in BigQuery, and then perform query export to GCS regularly (again with a Cloud Scheduler + Cloud Functions/Run). It's my preferred solution, because, maybe a day, you will have to process differently your message, and to get metrics/perform analytics on them.
0
0

@guillaume's answer is definitely the best one, but for ease of implement-ability, I decided to just save them directly to GCS.

const saveData = async ({ text, text2, enum, number }) => {
  try {
      const timestamp = new Date().getTime()
      const folder = enum
      const fileName = `${folder}/${enum}-${timestamp}.json`
      const file = bucket.file(fileName)
      const contents = JSON.stringify({ text, text2, enum, number })
      return file.save(contents)
    }
  } catch (e) {
    console.log(`Failed to save file, ${e.message}`)
  }
}

It added some latency, but overall I estimated the cost to be about $10 in server costs a month as compared to pubsub method which when trying to determine the cost, put it around $50-100 bucks a month (or more, was hard to determine. But it did assume that each message is 1MB if it's under 1MB).

The big query method Guillaume provided appeared to have no cost since 1TB of transferred data is free each month. I could be wrong on this. I may switch to this later on.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy