3

I'm currently learning how to use Capistrano to deploy a Rails app for the first time. I see the default flow of capistrano/rails includes running db:migrate on the host right before "publishing" the new symlink. However, there are some timing issues that concern me when running a deploy across multiple web nodes at the same time.

  1. If you migrate the database while the old version of the app is still running, you'll have old code talking to a changed DB which can lead to undefined behavior.
  2. If you turn off the app before running the migration, but the deploys go at less than identical speeds on each node, then one node could still have the old app running while the other node is already running the migrations.
  3. If two nodes invoke db:migrate at the same time, can't you run into a race condition from the fact they'll both think the new migrations haven't been run yet (by consulting the table), and then both try to run the migrations at the same time?

It seems to me the only sane solution is try rework the default flow into the following three stages:

  1. Normal deploy flow on each node up to but not including the db:migrate task -- so now the new version is installed, bundler ran, assets compiled, links made, everything ready (but not published via current symlink yet). Additional step at the end: stop the old app.
  2. After all hosts have completed stage 1, then run db:migrate on only one node (perhaps chosen at random, or just first on the list).
  3. After that's done, resume the second half of the normal flow on all hosts, and add a task for starting up the app again after the deploy:published task.

I'm just a little lacking in confidence in my own reasoning right now because while I've searched docs and googled for posts, I haven't really seen anyone talking about these multi-node issues, even though I have to assume it effects almost everyone...

1 Answer 1

0

You don't run rake db:migrate on multiple nodes. You run it on only 1 node. Remember, db migrate migrates the database, which is shared by all nodes in the app, so running it on different nodes at the same time is definitely a bad idea.

I split my db migrations from my data migrations to get quicker deploys, and then I do two different kinds of deploys: 1) With db migrations and some amount of downtime for the whole app 2) Without db migrations, and no-downtime (new dynos spin up while old dynos are running, after they boot up the router switches new traffic to new dynos, then shut down old dynos)

(Of course I'm doing in the context of Heroku which is different but same basic idea)

Remember also that it is absolutely necessary to re-boot Rails all your nodes after you run db:migrate, so your capistration script should account for that.

1
  • 1
    Thanks to some help from IRC and digging into the source, I've discovered that the deploy:migrate task only runs on the primary node of the group of hosts in the role defined by :migration_role. So there's also built-in support for handling that issue. I just need to add tasks before and after than stop/start the app on all web nodes, and I think I'll be ok.
    – odigity
    Jan 15, 2015 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.