I'm attempting to run 2 instances of Kafka on the same host. In order to achieve this, I download and extract Kafka and create a copy of the extracted Kafka folder and contents, creating two Kafka instances: Kafka-instance-1 and Kafka-instance-2.
I do not want to create a multiple broker setup within the same cluster, just 1 broker within each Kafka instance.
I update server.properties
on Kafka-instance-1 & Kafka-instance-2 to each point at different ports. So instead of the default port 9092, I use 8182. As I've started working on this I'm wondering if this is possible in Kafka - running two instances on the same machines, again to reiterate I'm not attempting to run two brokers but two independent Kafka instances. Are there pitfalls to attempting this? One thing that concerns me is that perhaps Kafka is using ports that are not exposed via config which will prevent multiple instances executing on the same machine.
Will this or any other problems be encountered trying to set up two Kafka instances on the same host ?
The reason I'm doing this is that I'm testing an app. The app requires its own set of topics. To test multiple instances of the app at the same time each app instance requires it's topic's to not be written to by other running instances of the app. For this reason, each app will have it's own Kafka instance. I could install Kafka on multiple machines and point each app to its own machine but that is extra work and is far easier to just create multiple Kafka instances on the same machine, that is if it's doable.