1

So my employer has Avro messages going through Kafka. We want to archive all messages to Amazon S3. Ideally, they would be stored in S3-like-directories by day and use a path structure something like:

s3://my-bucket/data/day=2016-03-04/data.avro

Is there a reference or best practices for how to do this well?

My one question is idempotency: How do I provide write idempotency, where a record may be sent to my sink writer more than once yet only be stored on S3 once.

Am I correct that I need idempotency? If I implement a simple append (non-idempotent), Kafka Connect may send the same records twice and they may get stored redundantly?

3 Answers 3

6

AFAIK, you cannot append to an S3 object (file): https://news.ycombinator.com/item?id=10746969 - unless something changed very recently....

You also do not want to keep writing each event into S3 individually, but rather do it in batches - for obvious performance reasons.

So, the way we do it - is by using Cassandra as an intermediate storage to collect events for some time period (timeslice) - store them grouped by event times , not processing times - and then write those timeslices that were touched into S3 periodically. One timeslice would be stored into one S3 object - with the timeslice info being part of the file name.

If/when more events come for some past timeslice - it is added to that timeslice table in Cassandra, and a re-write into S3 is [eventually] triggered - which would again get all events for that timeslice and write into S3 with the same file name, effectively overwriting the existing file if any.

You have to decide how long you want to keep data in Cassandra - based on how your pipeline works and how "old" your incoming events might be.

This is how we achieve idempotency. It's probably not the most efficient way - but it works well for us, since we have very high event processing volumes and rates and Cassandra is great for fast writes.

I'd love to hear how others solve similar problems!

3

Why not use something like secor? It also has some guarantees for exactly once delivery.

You could also consider "kafka connect" based implementations like streamx.

1

From the description, it looks like what you are looking for is 1) Avro data write to S3

2) Data to be partitioned in S3

3) Exactly-once support while writing.

Qubole StreamX supports rich variety of format conversions, avro being one of them, along with data partition. And, exactly once is in our pipeline which will be out soon.

Whereas secor is getting deprecated(mentioned in one of their responses on google group) and it also do not support avro.

So you can use qubole streamx to start with.

1
  • I've heard StreamX is also being deprecated, or at least not maintained or updated Sep 6, 2018 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.