0

I am using elasticsearch connector (confluent) for kafka connect. I am getting messages from topic into elasticsearch index. I can see that offset is part of _id if no key is used. I want to browse the messages in elasticseach based on offsets and timestamp.

Is it possible to get the offset and timestamp of message as fields in the index?

Here is the elasticsearch document

{ "_index": "test-elasticsearch-sink", "_type": "kafka-connect", "_id": "test-elasticsearch-sink+0+0", "_score": 1, "_source": { "f1": "value1" } }

Thanks, Rajesh

2 Answers 2

0

This is not currently possible, but we are thinking along the lines of a feature that will allow hoisting record metadata like timestamp, topic, partition, offset to fields.

1
  • Great! Thanks shikhar.
    – Rajesh
    Commented Nov 18, 2016 at 0:31
0

Using Single Message Transforms you can indeed add new fields including values such as offset and timestamp. Have a look at org.apache.kafka.connect.transforms.InsertField.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.