1

I want to set up a 3 node Rabbit cluster on EC2 (amazon linux). We'd like to have recovery implemented so if we lose a server it can be replaced by another new server automagically. We can set the cluster up manually easily using the default hostname (ip-xx-xx-xx-xx) so that the broker id is rabbit@ip-xx-xx-xx-xx. This is because the hostname is resolvable over the network.

The problem is: This hostname will change if we lose/reboot a server, invalidating the cluster. We haven't had luck in setting a custom static hostname because they are not resolvable by other machines in the cluster; thats the only part of that article that doens't make sense.

Has anyone accomplished a RabbitMQ Cluster on EC2 with a recovery implementation? Any advice is appreciated.

3 Answers 3

1

You could create three A records in an external DNS service for the three boxes and use them in the config. E.g., rabbit1.alph486.com, rabbit2.alph486.com and rabbit3.alph486.com. These could even be the ec2 private IP addresses. If all of the boxes are in the same region it'll be faster and cheaper. If you lose a box, just update the DNS record.

Additionally, you could assign an elastic IPs to the three boxes. Then, when you lose a box, all you'd need to do is assign the elastic IP to it's replacement.

Of course, if you have a small number of clients, you could just add entries into the /etc/hosts file on each box and update as needed.

1
  • Thanks for the response! Yes we did experiment with using hosts for this purpose and the thought of DNS. I wanted to try to avoid bolting on any new stuff (DNS) for the solution. The private IP peice may work but the default internal IPs can still change which is what the host is on the machine, so it didn't really help us. hosts file worked but was a little hacky since we'd have to query amazon and still write to the file locally on launch. Good answers though, confirms my suspicions on what's required!
    – alph486
    Feb 8, 2013 at 15:16
0

From:

http://www.rabbitmq.com/ec2.html

Issues with hostname

RabbitMQ names the database directory using the current hostname of the system. If the hostname changes, a new empty database is created. To avoid data loss it's crucial to set up a fixed and resolvable hostname. For example:

sudo -s # become root

echo "rabbit" > /etc/hostname

echo "127.0.0.1 rabbit" >> /etc/hosts

hostname -F /etc/hostname

0

@Chrskly gave good answers that are the general consensus of the Rabbit community:

  • Init scripts that handle DNS or identification of other servers are mainly what I hear.
  • Elastic IPs we could not get to work without the aid of DNS or hostname aliases because the Internal IP/DNS on amazon still rotate and the public IP/DNS names that stay static cannot be used as the hostname for rabbit unless aliased properly.
  • Hosts file manipulations via an script are also an option. This needs to be accompanied by a script that can identify the DNS's of the other servers upon launch so doesn't save much work in terms of making things more "solid state" config wise.

What I'm doing:

Due to some limitations on the DNS front, I am opting to use bootstrap scripts to initialize the machine and cluster with any other available machines using the default internal dns assigned at launch. If we lose a machine, a new one will come up, prepare rabbit and lookup the DNS names of machines to cluster with. It will then remove the dead node from the cluster for housekeeping.

I'm using some homebrew init scripts in Python. However, this could easily be done with something like Chef/Puppet.

Update: Detail from Docs

From:

http://www.rabbitmq.com/ec2.html

Issues with hostname

RabbitMQ names the database directory using the current hostname of the system. If the hostname changes, a new empty database is created. To avoid data loss it's crucial to set up a fixed and resolvable hostname. For example:

sudo -s # become root

echo "rabbit" > /etc/hostname

echo "127.0.0.1 rabbit" >> /etc/hosts

hostname -F /etc/hostname

2
  • Nice, glad you got what you were looking for.
    – chrskly
    Feb 8, 2013 at 16:42
  • Added some detail from @Amit 's answer re: how to script out a solution with hostnames.
    – alph486
    Jan 28, 2015 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.