2

Assume original message size is 500 bytes (before sending it to the Kafka). So what will be the size of the message after sending it to the Kafka? And what if we use any compression?

Additional information: I am putting a ByteBuffer of size 2048 bytes to a topic (with single partition) without any key.

Topic name: ub3
Path: /data/kafka-logs/ub3-0

[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
184
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M     00000000000000000000.index
448K    00000000000000000000.log
10M     00000000000000000000.timeindex
4.0K    leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$
[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
86284
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M     00000000000000000000.index
256M    00000000000000000000.log
10M     00000000000000000000.timeindex
4.0K    leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$


[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
172405
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M     00000000000000000000.index
512M    00000000000000000000.log
10M     00000000000000000000.timeindex
4.0K    leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$



[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
258491
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M     00000000000000000000.index
596M    00000000000000000000.log
10M     00000000000000000000.timeindex
4.0K    leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$



[hdpusr@hdpdev2 ub3-0]$ $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hdpdev2:8092 --topic ub3 --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
344563
[hdpusr@hdpdev2 ub3-0]$ du -sh *
10M     00000000000000000000.index
1.1G    00000000000000000000.log
10M     00000000000000000000.timeindex
4.0K    leader-epoch-checkpoint
[hdpusr@hdpdev2 ub3-0]$

enter image description here

1 Answer 1

5

The short answer is: who knows?

But let's try to find out some numbers. I have started a Kafka in Docker using this guide. Then, wrote a simple producer:

public class App {
    public static void main(String[] args) throws Exception {
        final Producer<String, byte[]> producer = producer();

        producer.send(
                new ProducerRecord<>(
                        "test",
                        key(),
                        value()
                )
        ).get();
    }

    private static Producer<String, byte[]> producer() {
        final Properties props = new Properties();

        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.CLIENT_ID_CONFIG, "so57472830");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());

        return new KafkaProducer<>(props);
    }

    private static String key() {
        return UUID.randomUUID().toString();
    }
}

So, will be sending to localhost:9092 with a client id equal to so57472830 into a test topic. The payloads are byte arrays and keys are string UUIDs. As you'll see later all these values (except the host:port) contribute to the "overhead". Here I suppose that overhead is everything except the message payload itself.

Let's start with a "Hello, world!":

private static byte[] value() {
    return "Hello, world!".getBytes();
}

Run the app and capture the traffic to localhost:9092. I used WireShark for that.

Here I found the message with the payload. Let's see the whole TCP stream ("Follow TCP stream" in WireShark):

So, the whole stream took 527 bytes of which the client send (highlighted with rose color) 195:

(This also means that Kafka send 527 - 195 == 332 bytes in response):

Our payload was 13 bytes. As you noticed, the outbound traffic contains the client id twice (2 × 10 bytes) and the message key (16 bytes). So, of 195 bytes send 146 are mystery (probably the one that you named as "overhead" in your question).

Let's send 500 random bytes:

private static byte[] value() {
    final byte[] result = new byte[500];

    new Random().nextBytes(result);

    return result;
}

Outbound traffic was 684 bytes (the entire conversation took 1016):

Again, the server send 332 byte in response and the outbound mystery (overhead) made up 684 - (500 + 2 × 10 + 16) = 164 bytes!

All these numbers are not final and may change with producer versions or specific config settings. One of them, you've mentioned, is compression. Let's check it out. Be warned that the compression depends on the data. Random bytes are tougher to compress than the constant ones as they have more entropy. So, let's send 500 repetiting bytes with a GZIP compression. Without the compression, the numbers are the same:

Add props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip"); to the producer() method and change the value():

private static byte[] value() {
    final byte[] result = new byte[500];

    Arrays.fill(result, (byte) 'a');

    return result;
}

When the compression is enabled, the message (key and value, not the client id and topic) are compressed, and the outbound traffic is only 208 bytes:

I'd say that the overhead is about the same as in the examples above, the compression impacts the size of the message itself.

That all applies to the traffic, but after your edit I see you were interested in the storage size. Nevertheless, I would say that the answer is the same: "who knows". The numbers definitely depend on your configuration.

1
  • Thanks for detailed explanation. Will wait for others to put their views, that may help. Aug 14, 2019 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.