Assume original message size is 500 bytes (before sending it to the Kafka). So what will be the size of the message after sending it to the Kafka? And what if we use any compression?
Additional information: I am putting a ByteBuffer of size 2048 bytes to a topic (with single partition) without any key.
Topic name: ub3
Path: /data/kafka-logs/ub3-0
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
184
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M 00000000000000000000.index
448K 00000000000000000000.log
10M 00000000000000000000.timeindex
4.0K leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
86284
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M 00000000000000000000.index
256M 00000000000000000000.log
10M 00000000000000000000.timeindex
4.0K leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
172405
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M 00000000000000000000.index
512M 00000000000000000000.log
10M 00000000000000000000.timeindex
4.0K leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
258491
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M 00000000000000000000.index
596M 00000000000000000000.log
10M 00000000000000000000.timeindex
4.0K leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
344563
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M 00000000000000000000.index
1.1G 00000000000000000000.log
10M 00000000000000000000.timeindex
4.0K leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$