19

I am using upstart to start/stop/automatically restart daemons. One of the daemons forks 4 times. The upstart cookbook states that it only supports forking twice. Is there a workaround?

How it fails

If I try to use expect daemon or expect fork, upstart uses the pid of the second fork. When I try to stop the job, nobody responds to upstarts SIGKILL signal and it hangs until you exhaust the pid space and loop back around. It gets worse if you add respawn. Upstart thinks the job died and immediately starts another one.

Bug acknowledged by upstream

A bug has been entered for upstart. The solutions presented are stick with the old sysvinit, rewrite your daemon, or wait for a re-write. RHEL is close to 2 years behind the latest upstart package, so by the time the rewrite is released and we get updated the wait will probably be 4 years. The daemon is written by a subcontractor of a subcontractor of a contractor so it will not be fixed any time soon either.

1 Answer 1

22

I came up with an ugly hack to make this work. It works for my application on my system. YMMV.

  1. start the application in the pre-start section
  2. in the script section run a script that runs as long as the application runs. The pid of this script is what upstart will track.
  3. in the post-stop section kill the application

example

env DAEMON=/usr/bin/forky-application

pre-start script
    su -s /bin/sh -c "$DAEMON" joeuseraccount
end script

script 
    sleepWhileAppIsUp(){
        while pidof $1 >/dev/null; do
            sleep 1
        done
    }

    sleepWhileAppIsUp $DAEMON
end script

post-stop script
    if pidof $DAEMON;
    then
        kill `pidof $DAEMON`
        #pkill  $DAEMON # post-stop process (19300) terminated with status 1
    fi
end script

a similar approach could be taken with pid files.

7
  • My service command is stuck. If this is the solution in the long term. How do I refresh my service command (to forget about the service that was forking too many times) short of restarting the server? Oct 31, 2013 at 16:16
  • @johnjamesmiller I'm currently using my script. It starts perfectly, but after it has started and I write status. It gives me: supervisord stop/waiting and it's not possible to use service supervisord stop. I thought it may have something to do with bugs.launchpad.net/upstart/+bug/406397/comments/47 but I'm not sure what he means. Oct 31, 2013 at 16:57
  • I fixed it, I basically replaced all mentions of pidof with pgrep. Seems like pidof wasn't working with me. Oct 31, 2013 at 17:18
  • @CMCDragonka if you do initctl log-priority debug you can see the jobs/services change status in /var/log/messages. If upstart is waiting on a pid that doesnt exist anymore then you need to exhaust your pid space so a new program pops up with the old pid. upstart will then kill that process and go back to a sane state. Creating a bunch of sleeps works well for this. Nov 6, 2013 at 16:31
  • 1
    In order to make this work with respawn (so the service keeps running even if it was killed or has crashed), all it took was adding an exit with non-zero code at the end of the main script section. This tells Upstart this was an abnormal exit and the daemon needs respawning. For this to work it's important that the post-stop script finishes cleanly; so indeed a brute-force pkill won't do, as suggested by comments.
    – sxc731
    Jul 21, 2016 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.