184

I'm trying to set up a cron job as a sort of watchdog for a daemon that I've created. If the daemon errors out and fails, I want the cron job to periodically restart it... I'm not sure how possible this is, but I read through a couple of cron tutorials and couldn't find anything that would do what I'm looking for...

My daemon gets started from a shell script, so I'm really just looking for a way to run a cron job ONLY if the previous run of that job isn't still running.

I found this post, which did provide a solution for what I'm trying to do using lock files, not I'm not sure if there is a better way to do it...

16 Answers 16

185

Use flock. It's new. It's better.

Now you don't have to write the code yourself. Check out more reasons here: https://serverfault.com/a/82863

/usr/bin/flock -n /tmp/my.lockfile /usr/local/bin/my_script
7
  • 1
    I also created a nice cron template gist here: gist.github.com/jesslilly/315132a59f749c11b7c6
    – Jess
    Commented Jan 30, 2017 at 12:54
  • 3
    setlock, s6-setlock, chpst, and runlock in their non-blocking modes are alternatives that are available on more than just Linux. unix.stackexchange.com/a/475580/5132
    – JdeBP
    Commented Oct 30, 2018 at 18:39
  • Does flock only with with scripts or will it also work with binaries like ansible-pull?
    – majorgear
    Commented Feb 12, 2023 at 16:54
  • 1
    @majorgear it will execute anything, but usually I see people running a script so they can set environment.
    – Jess
    Commented Feb 14, 2023 at 13:05
  • Thanks. I have Ansible-pull running every 10 minutes to keep my server configurations up to date. I added "flock -on /tmp/ansible-pull.lock " in front of the command and it's working to prevent duplicate instances from running.
    – majorgear
    Commented Feb 18, 2023 at 3:46
129

I do this for a print spooler program that I wrote, it's just a shell script:

#!/bin/sh
if ps -ef | grep -v grep | grep doctype.php ; then
        exit 0
else
        /home/user/bin/doctype.php >> /home/user/bin/spooler.log &
        #mailing program
        /home/user/bin/simplemail.php "Print spooler was not running...  Restarted." 
        exit 0
fi

It runs every two minutes and is quite effective. I have it email me with special information if for some reason the process is not running.

5
  • 4
    not a very safe solution, though, what if there are other process that matches the search you did in grep? rsanden's answer prevents that sort of problem using a pidfile. Commented Oct 6, 2012 at 3:24
  • 17
    This wheel was already invented somewhere else :) E.g., serverfault.com/a/82863/108394 Commented Jun 8, 2013 at 2:27
  • 5
    Instead of grep -v grep | grep doctype.php you can do grep [d]octype.php.
    – AlexT
    Commented Mar 16, 2016 at 9:23
  • Note that there's no need for the & if it is cron running the script.
    – lainatnavi
    Commented Nov 28, 2019 at 13:54
  • This is an EXCELLENT solution. Works like a charm and if you include the fully qualified path in the program name grep searches for, then it is unique. a minor improvement may be to add --quiet option on the rightmost grep.
    – Rahav
    Commented Oct 20, 2023 at 20:04
66

As others have stated, writing and checking a PID file is a good solution. Here's my bash implementation:

#!/bin/bash

mkdir -p "$HOME/tmp"
PIDFILE="$HOME/tmp/myprogram.pid"

if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |
                           grep -P "^\s*$(cat ${PIDFILE})$" &> /dev/null); then
  echo "Already running."
  exit 99
fi

/path/to/myprogram > $HOME/tmp/myprogram.log &

echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"
10
  • 3
    +1 Using a pidfile it's probably much safer than grepping for a running program with the same name. Commented Oct 6, 2012 at 3:22
  • /path/to/myprogram & > $HOME/tmp/myprogram.log & ?????? did you perhaps mean /path/to/myprogram >> $HOME/tmp/myprogram.log &
    – matteo
    Commented Nov 13, 2012 at 19:31
  • 1
    Shouldn't the file be removed when the script finishes? Or am I missing something very obvious?
    – Hamzahfrq
    Commented Jul 19, 2014 at 17:05
  • 1
    @matteo: Yes, you're right. I had fixed that in my notes years ago but forgot to update it here. Worse, I missed it in your comment too, noticing only ">" versus ">>". Sorry about that.
    – rsanden
    Commented Jul 23, 2014 at 0:05
  • 5
    @Hamzahfrq: Here's how it works: The script first checks to see if the PID file exists ("[ -e "${PIDFILE}" ]". If it does not, then it will start the program in the background, write its PID to a file ("echo $! > "${PIDFILE}""), and exit. If the PID file instead does exist, then the script will check your own processes ("ps -u $(whoami) -opid=") and see if you're running one with the same PID ("grep -P "^\s*$(cat ${PIDFILE})$""). If you're not, then it will start the program as before, overwrite the PID file with the new PID, and exit. I see no reason to modify the script; do you?
    – rsanden
    Commented Jul 23, 2014 at 0:07
60

It's suprising that no one mentioned about run-one. I've solved my problem with this.

 apt-get install run-one

then add run-one before your crontab script

*/20 * * * * * run-one python /script/to/run/awesome.py

Check out this askubuntu SE answer. You can find link to a detailed information there as well.

3
  • 4
    What's important to mention is that this tool is available out of the box in Ubuntu 20 (and maybe the versions prior as well)
    – Adam Sibik
    Commented Dec 26, 2020 at 10:22
  • This was a huge help
    – kokko1G
    Commented Jul 26, 2021 at 10:46
  • A portable rewrite for Debian, BSD and likely other *nix as well is available from here github.com/Freaky/run-one
    – korkman
    Commented Feb 17, 2022 at 21:26
28

Don't try to do it via cron. Have cron run a script no matter what, and then have the script decide if the program is running and start it if necessary (note you can use Ruby or Python or your favorite scripting language to do this)

1
  • 7
    The classic way is to read a PID file that the service creates when it starts, check if the process with that PID is still running, and restart if not.
    – tvanfosson
    Commented Mar 2, 2010 at 21:02
10

The way I am doing it when I am running php scripts is:

The crontab:

* * * * * php /path/to/php/script.php &

The php code:

<?php
if (shell_exec('ps aux | grep ' . __FILE__ . ' | wc  -l') > 1) {
    exit('already running...');
}
// do stuff

This command is searching in the system process list for the current php filename if it exists the line counter (wc -l) will be greater then one because the search command itself containing the filename

so if you running php crons add the above code to the start of your php code and it will run only once.

1
  • This is what I needed as all the other solutions required installing something on the client server, which I don't have access to do.
    – Jeff Davis
    Commented Jan 19, 2017 at 22:26
9

You can also do it as a one-liner directly in your crontab:

* * * * * [ `ps -ef|grep -v grep|grep <command>` -eq 0 ] && <command>
5
  • 5
    not very safe, what if there are other commands that matches the search for grep? Commented Oct 6, 2012 at 3:22
  • 1
    This could also be written as * * * * * [ ps -ef|grep [c]ommand -eq 0 ] && <command> where wrapping the first letter of your command in brackets excludes it from the grep results.
    – Jim Clouse
    Commented Feb 5, 2013 at 19:01
  • I had to use the following syntax: [ "$(ps -ef|grep [c]ommand|wc -l)" -eq 0 ] && <command>
    – thameera
    Commented Mar 24, 2016 at 19:36
  • 3
    This is hideous. [ $(grep something | wc -l) -eq 0 ] is a really roundabout way to write ! grep -q something. So you want simply ps -ef | grep '[c]ommand' || command
    – tripleee
    Commented Sep 19, 2016 at 11:16
  • (Also, as an aside, if you really wanted to actually count the number of matching lines, that's grep -c.)
    – tripleee
    Commented Sep 19, 2016 at 11:17
6

As a follow up to Earlz answer, you need a wrapper script that creates a $PID.running file when it starts, and delete when it ends. The wrapper script calls the script you wish to run. The wrapper is necessary in case the target script fails or errors out, the pid file gets deleted..

1
  • Oh cool... I never thought about using a wrapper... I couldn't figure out a way to do it using lock files because I couldn't guarantee that the file would get deleted if the daemon errored out... A wrapper would work perfectly, I'm going to give jjclarkson's solution a shot, but I'll do this if that doesn't work...
    – LorenVS
    Commented Mar 2, 2010 at 21:06
5

With lockrun you don't need to write a wrapper script for your cron job. http://www.unixwiz.net/tools/lockrun.html

4
# one instance only (works unless your cmd has 'grep' in it)
ALREADY_RUNNING_EXIT_STATUS=0
bn=`basename $0`
proc=`ps -ef | grep -v grep | grep "$bn" | grep -v " $$ "`
[ $? -eq 0 ] && {
    pid=`echo $proc | awk '{print $2}'`
    echo "$bn already running with pid $pid"
    exit $ALREADY_RUNNING_EXIT_STATUS
}

UPDATE .. better way using flock:

/usr/bin/flock -n /tmp/your-app.lock /path/your-app args 
3

I would recommend to use an existing tool such as monit, it will monitor and auto restart processes. There is more information available here. It should be easily available in most distributions.

1
  • Every answer except this one answers the surface question "How can my cron job make sure it only runs one instance?" when the real question is "How can I keep my process running in the face of restarts?", and the correct answer is indeed to not use cron, but instead a process supervisor like monit. Other options include runit, s6 or, if your distribution already uses systemd, just creating a systemd service for the process that needs to be kept alive.
    – clacke
    Commented Dec 31, 2018 at 4:01
2

This one never failed me:

one.sh:

LFILE=/tmp/one-`echo "$@" | md5sum | cut -d\  -f1`.pid
if [ -e ${LFILE} ] && kill -0 `cat ${LFILE}`; then
   exit
fi

trap "rm -f ${LFILE}; exit" INT TERM EXIT
echo $$ > ${LFILE}

$@

rm -f ${LFILE}

cron job:

* * * * * /path/to/one.sh <command>
2

Docs: https://github.com/timkay/solo

solo is a very simple script (10 lines) that prevents a program from running more than one copy at a time. It is useful with cron to make sure that a job doesn't run before a previous one has finished.

Example

* * * * * solo -port=3801 ./job.pl blah blah
2
  • 1
    This should be higher in the list. It avoids all the crap that lockfiles involve (stale files, lock reliability that depends on the underlying filesystem (think NFS), race conditions due to non-atomic naïve implementations, and so on). Commented Mar 13, 2023 at 7:51
  • The link above to the solo script seems to be broken.
    – Steve
    Commented Apr 30 at 16:55
1

I'd suggest the following as an improvement to rsanden's answer (I'd post as a comment, but don't have enough reputation...):

#!/usr/bin/env bash

PIDFILE="$HOME/tmp/myprogram.pid"

if [ -e "${PIDFILE}" ] && (ps -p $(cat ${PIDFILE}) > /dev/null); then
  echo "Already running."
  exit 99
fi

/path/to/myprogram

This avoids possible false matches (and the overhead of grepping), and it suppresses output and relies only on exit status of ps.

1
  • 1
    Your ps command would match PIDs for other users on the system, not just your own. Adding a "-u" to the ps command changes the way the exit status works.
    – rsanden
    Commented Jul 21, 2014 at 0:15
1

Simple custom php is enough to achieve. No need to confuse with shell script.

lets assume you want to run php /home/mypath/example.php if not running

Then use following custom php script to do the same job.

create following /home/mypath/forever.php

<?php
    $cmd = $argv[1];
    $grep = "ps -ef | grep '".$cmd."'";
    exec($grep,$out);
    if(count($out)<5){
        $cmd .= ' > /dev/null 2>/dev/null &';
        exec($cmd,$out);
        print_r($out);
    }
?>

Then in your cron add following

* * * * * php /home/mypath/forever.php 'php /home/mypath/example.php'
0

Consider using pgrep (if available) rather than ps piped through grep if you're going to go that route. Though, personally, I've got a lot of mileage out of scripts of the form

while(1){
  call script_that_must_run
  sleep 5
}

Though this can fail and cron jobs are often the best way for essential stuff. Just another alternative.

1
  • 2
    This would just start the daemon over and over again and does not solve the problem mentioned above.
    – cwoebker
    Commented May 5, 2013 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.