3

does anyone know how to lock on a function in bash script? I wanted to do something like in java (like synchronize), ensuring that each file saved in monitored folder is on hold ever tries to use submit function.

an excerpt from my script:

(...)

ON_EVENT () { local date = $1 local time = $2 local file = $3 sleep 5 echo "$date $time New file created: $file" submit $file }

submit () { local file = $1 python avsubmit.py -f $file -v python dbmgr.py -a $file }

if [ ! -e "$FIFO" ]; then mkfifo "$FIFO" fi

inotifywait -m -e "$EVENTS" --timefmt '%Y-%m-%d %H:%M:%S' --format '%T %f' "$DIR" > "$FIFO" & INOTIFY_PID=$!

trap "on_exit" 2 3 15

while read date time file do on_event $date $time $file & done < "$FIFO"

on_exit

I'm using inotify to monitor a folder when a new file is saved. For each file saved (received), submit to VirusTotal service (avsubmit.py) and TreathExpert (dbmgr.py). Concurrent access would be ideal to avoid blocking every new file created in monitored folder, but lock submit function should be sufficient.

Thank you guys!

6 Answers 6

6

Something like this should work:

if (set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; then
   trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
   # Your code here
   rm -f "$lockfile"
   trap - INT TERM EXIT
else
   echo "Failed to acquire $lockfile. Held by $(cat $lockfile)"
fi
0
3

Any code using rm in combination with trap or similar facility is inherently flawed against ungraceful kills, panics, system crashes, newbie sysadmins, etc. The flaw is that the lock needs to be manually cleaned after such catastrophic event for the script to run again. That may or may not be a problem for you. It is a problem for those managing many machines or wishing to have an unplugged vacation once in a while.

A modern solution using a file descriptor lock has been around for a while - I detailed it here and a working example is on the GitHub here. If you do not need to track process ID for whatever monitoring or other reasons, there is an interesting suggestion for a self-lock (I did not try it, not sure of its portability guarantee).

1
  • Thanks. I just used a version of your flock solution. Works great.
    – DoxyLover
    Commented Feb 19, 2014 at 20:15
1

You can use a lock file to determine whether or not the file should be submitted.

Inside your ON_EVENT function, you should check if the appropriate lock file exists before calling the submit function. If it does exist, then return, or sleep and check again later to see if it's gone. If it doesn't exist, then create the lock and call submit. After the submit function completes, then delete the lock file.

See this thread for implementation details.

0

But I liked that files can not get lock stay on the waiting list (cache) to be submitted then or later.

I currently have something like this:

lockfile="./lock"

on_event() { local date=$1 local time=$2 local file=$3 sleep 5 echo "$date $time New file created: $file" if (set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; then trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT submit_samples $file rm -f "$lockfile" trap - INT TERM EXIT else
echo "Failed to acquire lockfile: $lockfile." echo "Held by $(cat $lockfile)" fi }

submit_samples() { local file=$1 python avsubmit.py -f $file -v python dbmgr.py -a $file }

Thank you once again ...

0
0

I had proplems wiith this approach and found a better solution:

Procmail comes with a lockfile command which does what I wanted:

lockfile -5 -r10 /tmp/lock.file
do something very important
rm -f /tmp/lock.file

lockfile will try to create the specified lockfile. If it exists it iwll retry in 5 seconds, this will be repeated for maximum 10 times. If can create the flile it goes on with the script.

Another solution are the lockfile-progs in debian, example directly from the man page:

Locking a file during a lengthy process:

     lockfile-create /some/file
     lockfile-touch /some/file &
     # Save the PID of the lockfile-touch process
     BADGER="$!"
     do-something-important-with /some/file
     kill "${BADGER}"
     lockfile-remove /some/file
0

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

inotifywait -q -m -r -e CLOSE_WRITE --format %w%f $DIR | 
parallel -u python avsubmit.py -f {}\; python dbmgr.py -a {}

It will run at most one python per CPU when a file is written (and closed). That way you can bypass all the locking, and you get the added benefit that you avoid a potential race condition where a file is immediately overwritten (how do you make sure that both the first and the second version was checked?).

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.