14

One target in my makefile is a very CPU and time consuming task. But I can split the workload and run the task several times in parallel to speed up the entire process.

My problem is that make doesn't wait for all processes to complete.

Consider this simple script, named myTask.sh:

#!/bin/bash

echo "Sleeping $1 seconds"
sleep $1
echo "$1 are over!"

Now, let's call this from a bash script, and use wait to wait for all tasks to complete:

#!/bin/bash

echo "START"
./myTask.sh 5 &
./myTask.sh 15 &
./myTask.sh 10 &

wait  # Wait for all tasks to complete

echo "DONE"

The output is as expected:

START
Sleeping 15 seconds
Sleeping 5 seconds
Sleeping 10 seconds
5 are over!
10 are over!
15 are over!
DONE

But when trying the same in a Makefile:

test:
    echo "START"
    ./myTask.sh 5 &
    ./myTask.sh 15 &
    ./myTask.sh 10 &
    wait
    echo "DONE"

it doesn't work:

START
Sleeping 5 seconds
Sleeping 15 seconds
Sleeping 10 seconds
DONE
sweber@pc:~/testwait $5 are over!
10 are over!
15 are over!

Of course, I could create multiple targets which can be "built" in parallel by make, or let make just run the bash script which runs the tasks. But is there a way to do it more like what I already tried?

3
  • My first attempt would be to put all relevant commands in the same line: ./myTask.sh 5 & ./myTask.sh 15 & ./myTask.sh 10 & wait
    – melpomene
    Commented Jan 3, 2017 at 21:32
  • 2
    Ideal job for GNU Parallel... parallel ./myTask.sh ::: 5 10 15 Commented Jan 3, 2017 at 22:00
  • If you are using GNU make, and the ONESHELL feature is installed, your script will work "as-is" if you add a line with .ONESHELL: at the start since that will force your entire recipe to run in a single invocation of your shell. Commented Jan 6, 2017 at 17:02

2 Answers 2

17

Why not use the mechanisms already built in make? Something like this:

task%:
        sh task.sh $*

test: task5 task15 task10
        echo "DONE"

You will be able to adjust the level of parallelism on the make command line, instead of having it hard-coded into your makefile (e.g use 'make -j 2 test' if you have 2 cores available and 'make -j 32 test' if you have 32 cores)

3
  • 1
    How do you attach parameters please? E.g. task 5 rather than task5. Commented Jan 4, 2017 at 0:47
  • 1
    @MarkSetchell what do you mean? task5 is the name of a target, it has to meet the syntactic requirements for a target. In the recipe for the task5 target, the stem $* will be 5 and you can do whatever you want with that (e.g. use it as a parameter for the task.sh script as shown in the example above)
    – Come Raczy
    Commented Jan 4, 2017 at 0:53
  • 1
    You can automate the level of parallelism with -j $(nproc) instead of using an integer constant. nproc outputs the number of processors.
    – doug65536
    Commented Aug 9, 2018 at 20:39
15

Each individual logical line in a recipe is invoked in its own shell. So your makefile is basically running this:

test:
        /bin/sh -c 'echo "START"'
        /bin/sh -c './myTask.sh 5 &'
        /bin/sh -c './myTask.sh 15 &'
        /bin/sh -c './myTask.sh 10 &'
        /bin/sh -c 'wait'
        /bin/sh -c 'echo "DONE"'

You should make it all run in a single shell using semicolons and backslash/newlines to combine the physical lines into one logical line:

test:
        echo "START"; \
        ./myTask.sh 5 & \
        ./myTask.sh 15 & \
        ./myTask.sh 10 & \
        wait; \
        echo "DONE"
4
  • & is already a command separator. You don't need an extra ;.
    – melpomene
    Commented Jan 3, 2017 at 21:37
  • Each individual logical line in a recipe is invoked in its own shell. That was the missing point. Thanks!
    – sweber
    Commented Jan 4, 2017 at 14:04
  • Is it possible to stop this task with Cntrl+C? It "kills" the makefile, but the task keeps running for me Commented Sep 9, 2020 at 19:33
  • You can but you'll have to do it yourself with a more complex script. There's nothing in make itself which will do this, because make has no idea what other processes are running so it can't kill them. You can write your script to use the shell's trap function to catch an interrupt signal and kill all the job(s) that are running in the background. Commented Sep 9, 2020 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.