2

I´m starting with Apache Kafka and i´m facing problems when i try to conect from an external machine.

With this configuration bellow, all works fine if the application and the docker are running at the same machine.

but when i put the application in machine A and docker at machine B, the application cant connect.

My spring Kafka @Configuration have this line to @Bean consumerFactory and producerFactory (imagine my machine with docker ip = 10.10.10.10)

props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.10.10.10:9092");

And my docker file is this:

version: '2'
services:
zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - 2181:2181
kafka:
    image: wurstmeister/kafka:0.10.1.1
    environment:
        KAFKA_ADVERTISED_HOST_NAME: 0.0.0.0
        KAFKA_ADVERTISED_PORT: 9092
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        KAFKA_CREATE_TOPICS: "topic-jhipster:1:1,PROCESS_ORDER:1:1, PROCESS_CHANNEL:1:1"
        JMX_PORT: 9999
        KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=9999"
    ports:
        - 9092:9092
        - 9999:9999
kafka-manager:
    image: sheepkiller/kafka-manager
    ports:
        - 9000:9000
    links:
        - zookeeper
    environment:
        ZK_HOSTS: zookeeper:2181

i get this error:

org.springframework.kafka.core.KafkaProducerException: Failed to send;
nested exception is org.apache.kafka.common.errors.TimeoutException: 
Expiring 1 record(s) for

Edit, add some information..

I think its any configuration about the zookeeper i´m missing .. because if i have only the zookeeper started at my machine A .. and the kafka in machine B.. that works.. i only don´t know how :(

2
  • 1
    Maybe a firewall? Jul 28, 2017 at 20:10
  • @EvgeniDimitrov not, thats all ok. I think its any configuration about the zookeper i´m missing .. because if i have only the zookeeper started at my machine A .. and the kafka in machine B.. that works.. i only don´t know how :(
    – fhgomes_ti
    Jul 28, 2017 at 20:37

4 Answers 4

1

Try setting Listeners,

eg: listeners = PLAINTEXT://your.host.name:9092

Assuming you can telnet between the machines on kafka port.

1

Set advertised.listeners to the hostname or ip of the docker container's host machine.

environment:
        KAFKA_ADVERTISED_LISTENERS: "10.10.10.10:9092"

What is happening is that the client connects to the bootstrap server, does a meta-data request to discover which Kafka broker in the cluster to connect to for a particular topic partition, and gets the advertised host name back as a response (in your case 0.0.0.0) which will only work if everything is on the same machine.

You need to advertise an ip or hostname that will work from the remote machine so not localhost, 127.0.0.1, or 0.0.0.0. Also not the private internal IP or hostname of the docker container. It has to be the external/public IP or hostname.

Also advertised.host.name and advertised.port are deprecated parameters in Kafka 0.10.x so even if you use them (they work for backward compatibility), you also need to set the advertised.host.name to be something that the producer can resolve and connect to. I suggest using a fully qualified host name or IP of the docker host machine (I.e. 10.10.10.10).

From http://kafka.apache.org/0101/documentation.html#brokerconfigs

DEPRECATED: only used when advertised.listeners or listeners are not set. Use advertised.listeners instead. Hostname to publish to ZooKeeper for clients to use. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, it will use the value for host.name if configured. Otherwise it will use the value returned from java.net.InetAddress.getCanonicalHostName().

1

Some mix of @Krishas and @Hans Jespersen

Here is the code of my docker yml:

version: '2'
services:
    zookeeper:
        image: wurstmeister/zookeeper:3.4.6
        ports:
          - 2181:2181
    kafka:
        image: wurstmeister/kafka:0.10.1.1
        environment:
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.10.10.10:9092
            KAFKA_ADVERTISED_HOST_NAME: 10.10.10.10
            KAFKA_ADVERTISED_PORT: 9092
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

That needs the "PLAINTEXT:// prefix ! And config the "host_name" + "port", or "listeners"

The next step is decovery how i will configure another nodes

0

You need to specify java.rmi.server.hostname=0.0.0.0

3
  • Hi, are u my relative (Gomes)? kkk Where u say i need to specify that ?
    – fhgomes_ti
    Jul 28, 2017 at 20:38
  • May be yes!The option you can found on KAFKA_JMX_OPTS variable Jul 28, 2017 at 20:42
  • thats only a config to the Kafka Manager access the Kafka.. I think my problem is some configuration i´m missing at my producerFactory related with zookeeper .. but i dont know what is
    – fhgomes_ti
    Jul 28, 2017 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.