Is it possible to produce a million records a second via kafka? How many servers should it take? Currently i am sending 7000 messages a second from a kafka producer and struggling to exceed this
I've noticed several sources say kafka can send millions of messages a second
I've created a job that uses @Autowired kafkaTemplate and makes a while loop that sends a short text string "asdf" I've set up linger to 1000ms and seeing messages group in sets of 7000. The producer , consumer broker, zookeeper are on the same machine and the broker and zookeeper and a very simple docker image with default configuration
I am maxing out around 7000 requests a second
application.props
spring.kafka.bootstrap-servers=PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093
host.name=localhost
Job to make calls
@Async
@Scheduled(fixedDelay = 15000)
public void scheduleTaskUsingCronExpression() {
generateCalls();
}
private void generateCalls() {
try{
int i = 0;
System.out.println("start");
long startTime = System.currentTimeMillis();
while(i <= 1000000){
String message = "Test Message sadg sad-";
kafkaTemplate.send(TOPIC, message + i);
i++;
}
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime));
System.out.println("done");
}
catch(Exception e){
e.printStackTrace();
}
System.out.println("RUNNING");
}
Kakfa partition config
@Bean
public KafkaAdmin kafkaAdmin() {
//String bootstrapAddress = "localhost:29092";
String bootstrapAddress = "localhost:9092";
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
@Bean
public NewTopic testTopic() {
return new NewTopic("test-topic", 6, (short) 1);
}
Kafka consumer consuming messge
@KafkaListener(topics = "test-topic", groupId = "one", concurrency = "6" )
public void listenGroupFoo(String message) {
if(message.indexOf("-0") != -1){
startTime = new Date().getTime();
System.out.println("Starting Message in group foo: " + message);
}
else if(message.indexOf("-100000") != -1){
endTime = new Date().getTime();
System.out.println("Received Message in group foo: " + message);
System.out.println(endTime - startTime);
}
}
For hardware I have a 10900k with 64gb ram 5ghz clock speed 970 Evo single nvme disk 10 core 20 thread
All requests are from the same machine to the same machine
Is there a better way to organize / optimize the code to make a massive number of requests? Theories:
- Multiple Threads?
- Changing configurations of servers such as tomcat configs (receiving or sending side)?
- Not use the kafkaTemplate that is autowired or creating multiple?
- Modify Hardware to have multiple disks?
- Not use a job for the producer?
- Anything else anyone can think of to help?
message
outside of the loop, and you also do a new String each time (which takes time as well). I would suggest try multiple threads using an Stream makes this quite easyIntStream.rangeClosed(0, 100000).parallel().mapToObject(i -> message + i).forEach(msg -> kafkaTemplate.send(TOPIC, msg));
. Also doing both sending/receiving on the same machine will also limit the through put due to sharing resources (cpu, disc) and that will be noticable.