I am trying to push the logs from filebeat to AWS MSK (having TLS client Authentication enabled using AWS ACM PCA) and getting below errors:
2020-07-21T14:18:00.751Z ERROR kafka/client.go:92 Kafka connect fails with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2020-07-21T14:18:00.751Z ERROR pipeline/output.go:100 Failed to connect to kafka(b-3:9094,b-1:9094,b-2:9094): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2020-07-21T14:18:00.751Z INFO pipeline/output.go:93 Attempting to reconnect to kafka with 46 reconnect attempt(s)
2020-07-21T14:18:00.751Z INFO kafka/log.go:53 kafka message: Initializing new client
2020-07-21T14:18:00.751Z INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker b-1:9094
2020-07-21T14:18:00.756Z INFO kafka/log.go:53 Failed to connect to broker b-1:9094: local error: tls: no renegotiation
2020-07-21T14:18:00.756Z INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *net.OpError=local error: tls: no renegotiation)
2020-07-21T14:18:00.756Z INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker b-3:9094
2020-07-21T14:18:00.762Z INFO kafka/log.go:53 Failed to connect to broker b-3:9094: local error: tls: no renegotiation
2020-07-21T14:18:00.762Z INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *net.OpError=local error: tls: no renegotiation)
2020-07-21T14:18:00.762Z INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker b-2:9094
2020-07-21T14:18:00.767Z INFO kafka/log.go:53 Failed to connect to broker b-2:9094: local error: tls: no renegotiation
2020-07-21T14:18:00.767Z INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *net.OpError=local error: tls: no renegotiation)
2020-07-21T14:18:00.767Z INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-07-21T14:18:00.767Z INFO kafka/log.go:53 client/brokers resurrecting 3 dead seed brokers
2020-07-21T14:18:00.767Z INFO kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
===Filebeat config :===
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/abc.log
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["b-3:9094", "b-1:9094", "b-2:9094"]
# message topic selection + partitioning
topic: file
ssl.enabled: true
tls:
certificate: "/home/ec2-user/keystore_nopassphrase.pem"
certificate_key: "/home/ec2-user/private_key.pem"
certificate_authorities:
- "/home/ec2-user/truststore.pem"
partition.round_robin:
reachable_only: false
required_acks: 1
======================
As filebeat using golang so i have converted jks keystore and truststore into .pem file however i am suspecting that this conversion is not working. Further i would like top add that i was able to push the logs from filebeat to AWS MSK with plaintext and TLS traffic (without client auth).
===== client.properties :
security.protocol=SSL
ssl.truststore.location=/home/ec2-user/kafka_2.12-2.2.1/kafka.client.truststore.jks
ssl.keystore.location=/home/ec2-user/kafka_2.12-2.2.1/kafka.client.keystore.jks
ssl.keystore.password=**************
ssl.key.password=***********
=====
Can someone give detailed explanation on what are the files I've to pass on to the below configurations in filebeat.yml and how to convert it correctly.
ssl.certificate_authorities
ssl.certificate
ssl.key
Thankyou.