0

I'm going to use Ansible script to create Kafka topics. I define a YAML Hash var for topics to create as:

topics:
  topic1:
    partitions: 1
    replication_factor: 2
  topic2:
    partitions: 1
    replication_factor: 2

Before create them, I need first check if a topic has existed. I can use kafka-topics.sh --list --zookeeper localhost:2181 to get the list of existing topic names, then compare with topics hash to get a list of topics to create, then call kafka-topics.sh --create --topic name to create topics.

How to implement this?

2 Answers 2

1

Thanks for linuxdynasty's answer. But I don't want to write a module just for this simple task. Finally I worked out the following solution. I'm posting here for people referencing.

---
- name: Get existing topic list
  become_user: root
  become: yes
  become_method: sudo
  shell: /opt/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181
  register: existings

- name: Print existing topic list
  debug: msg="{{existings.stdout | regex_replace("\\n", " ")}}"

- name: Create topics
  become_user: root
  become: yes
  become_method: sudo
  shell: >
    if [[ `echo "{{existings.stdout}}" | grep {{item.key}}` != {{item.key}} ]]; then
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181
    --topic {{item.key}} --partitions {{item.value.partitions}}
    --replication-factor {{item.value.replication_factor}};
    fi
  with_dict: "{{kafka_topics}}"
0

It seems like you will need to create a kafka facts module. The module would be called something like this kafaka_topics_facts. The module will return the topics and than from there you can create the topics that do not exist. Please read this Developing Ansible Modules

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.