Recently I came across a bottleneck in our ansible playbooks' code. We were deploying our clusters (e.g. a mongoDB Replica Set) sequentially - i.e. one VM after another, each waiting for the previous to be up and running.
This slowed down the whole cluster deploy time by a factor of the members on it.
To solve this, I started digging on ansible's async actions and pooling and found out a few examples on parallel loops and "fire-and-forget" strategies for scenarios like ours.
The particular thing is, we have defined our own "customize the VM and spawn it" ansible task (create_instance.yml
) that gets included and receives the different customization variables from the playbook and abstracts the whole process by running different KVM/shell commands.
Using "Parallel task execution in Ansible" as reference, I ended up having something like:
- name: Generate VMs for DB
hosts: hypervisor_fe
tags: platform,mongodb
tasks:
- include: tasks/create_instance.yml
vars:
vm: "{{ item }}"
with_items: "{{ mongodb.vms }}"
register: mongo_instances
async: 7200
poll: 0
- name: Wait for instance creation to complete
async_status: jid={{ item.ansible_job_id }}
register: mongo_jobs
until: mongo_jobs.finished
retries: 300
with_items: "{{ mongo_instances.results }}"
However, this setup does seem to ignore all the new async code and keeps the old, sequential behavior. I'm guessing this has to do with the no. and granularity of plays inside the imported task. If I instead replace the include
for a single, explicit long-running task - let's say, e.g.
- name: Test async operation
shell: ping -c1 {{ item.hostname }} && sleep 20
This does seem to work just fine, running one ping to each item
and then moving on to the next action.
Is this assumption right? Does someone has experience with include
and async loops in ansible? Do I need to move the async declaration to a single play inside the imported code?