94

I was using SLURM to use some computing cluster and it had the -ntasks or -n. I have obviously read the documentation for it (http://slurm.schedmd.com/sbatch.html):

sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

the specific part I do not understand what it means is:

run within the allocation will launch a maximum of number tasks and to provide for sufficient resources.

I have a few questions:

  1. I guess my first question is what does the word "task" mean and the difference is with the word "job" in the SLURM context. I usually think of a job as the running the bash script under sbatch as in sbatch my_batch_job.sh. Not sure what task means.
  2. If I equate the word task with job then I thought it would have ran the same identical bash script multiple times according to the argument to -n, --ntasks=<number>. However, I obviously tested it out in the cluster, ran a echo hello with --ntask=9 and I expected sbatch would echo hello 9 times to stdout (which is collected in slurm-job_id.out, but to my surprise, there was a single execution of my echo hello script Then what does this command even do? It seems it does nothing or at least I can't see whats suppose to be doing.

I do know the -a, --array=<indexes> option exists for multiple jobs. That is a different topic. I simply want to know what --ntasks is suppose to do, ideally with an example so that I can test it out in the cluster.

3 Answers 3

88

The --ntasks parameter is useful if you have commands that you want to run in parallel within the same batch script. This may be two separate commands separated by an & or two commands used in a bash pipe (|).

For example

Using the default ntasks=1

#!/bin/bash

#SBATCH --ntasks=1

srun sleep 10 & 
srun sleep 12 &
wait

Will throw the warning:

Job step creation temporarily disabled, retrying

The number of tasks by default was specified to one, and therefore the second task cannot start until the first task has finished. This job will finish in around 22 seconds. To break this down:

sacct -j515058 --format=JobID,Start,End,Elapsed,NCPUS

        JobID               Start                 End    Elapsed      NCPUS
------------ ------------------- ------------------- ---------- ----------
515058       2018-12-13T20:51:44 2018-12-13T20:52:06   00:00:22          1
515058.batch 2018-12-13T20:51:44 2018-12-13T20:52:06   00:00:22          1
515058.0     2018-12-13T20:51:44 2018-12-13T20:51:56   00:00:12          1
515058.1     2018-12-13T20:51:56 2018-12-13T20:52:06   00:00:10          1

Here task 0 started and finished (in 12 seconds) followed by task 1 (in 10 seconds). To make a total user time of 22 seconds.

To run both of these commands simultaneously:

#!/bin/bash

#SBATCH --ntasks=2

srun --ntasks=1 sleep 10 & 
srun --ntasks=1 sleep 12 &
wait

Running the same sacct command as specified above

    sacct -j 515064 --format=JobID,Start,End,Elapsed,NCPUS
    JobID               Start                 End    Elapsed      NCPUS
    ------------ ------------------- ------------------- ---------- ----------
    515064       2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          2
    515064.batch 2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          2
    515064.0     2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          1
    515064.1     2018-12-13T21:34:08 2018-12-13T21:34:18   00:00:10          1

Here the total job taking 12 seconds. There is no risk of jobs waiting for resources as the number of tasks has been specified in the batch script and therefore the job has the resources to run this many commands at once.

Each task inherits the parameters specified for the batch script. This is why --ntasks=1 needs to be specified for each srun task, otherwise each task uses --ntasks=2 and so the second command will not run until the first task has finished.

Another caveat of the tasks inheriting the batch parameters is if --export=NONE is specified as a batch parameter. In this case --export=ALL should be specified for each srun command otherwise environment variables set within the sbatch script are not inherited by the srun command.

Additional notes:
When using bash pipes, it may be necessary to specify --nodes=1 to prevent commands either side of the pipes running on separate nodes.
When using & to run commands simultaneously, the wait is vital. In this case, without the wait command, task 0 would cancel itself, given task 1 completed successfully.

9
  • 3
    It seems that sometimes also --exclusive is needed for srun, as detailed in this answer on SO. It has two different meanings. When starting a new job, it uses the node exclusively. But if running job steps (srun in a batch script), it allows to run the job steps concurrently: "This option can also be used when initiating more than one job step within an existing resource allocation ..., where you want separate processors to be dedicated to each job step." (from the docs)
    – akraf
    Commented Jul 12, 2021 at 13:14
  • 1
    Thanks for looking in to this. Would the --exact parameter be more appropriate? From slurm.schedmd.com/news.html -- "In conjunction to this non-overlapping step allocation behavior being the new default, there is an additional new option for step management '--exact', which will allow a step access to only those resources requested by the step." I've also found this fix in version 20.11.6 - I wonder if either of these parameters are required in a newer release? github.com/SchedMD/slurm/commit/… Commented Jul 15, 2021 at 9:04
  • 1
    Unfortunately, I am very new to SLURM and don't really understand it yet. So I cannot answer that. Maybe a new question like "Did the way to start new parallel jobs change in SLURM" or similar could help?
    – akraf
    Commented Jul 15, 2021 at 18:40
  • 2
    Why do you use srun inside the batch script? Can I not just use e.g. eval? What is the advantage of using srun there?
    – Paloha
    Commented Jul 24, 2022 at 0:42
  • 4
    That's a good question Paloha, srun inside sbatch can give you control over cpu and memory allocation of individual tasks inside the batch call. Otherwise if each process in the batch call is being run sequentially then it's not needed. srun will produce its own metrics for the task though so this way you can collect mem and cpu metrics per process rather than just one for the entire batch call Commented Aug 12, 2022 at 2:39
36

The "--ntasks" options specifies how many instances of your command are executed. For a common cluster setup and if you start your command with "srun" this corresponds to the number of MPI ranks.

In contrast the option "--cpus-per-task" specify how many CPUs each task can use.

Your output surprises me as well. Have you launched your command in the script or via srun? Does you script look like:

#!/bin/bash
#SBATCH --ntasks=8
## more options
echo hello

This should always output only a single line, because the script is only executed on the submitting node not the worker.

If your script look like

#!/bin/bash
#SBATCH --ntasks=8
## more options
srun echo hello

srun causes the script to run your command on the worker nodes and as a result you should get 8 lines of hello.

4
  • Hey would you mind taking a look at my follow up question here: stackoverflow.com/questions/53952488/…? It's about how to allocate resources when running jobs in parallel like so. Commented Dec 28, 2018 at 6:05
  • 9
    This answer contradicts the one by @Alexis Lucattini Commented Sep 21, 2020 at 1:09
  • 1
    Not necessarily @aaronsnoswell. I have not tried but srun might implicitly run a task <ntasks> times if the '--n-tasks'-switch is not explicitly stated. Commented Apr 30, 2022 at 6:23
  • @aaronsnoswell according to this documentation (doc.cc.in2p3.fr/en/Computing/slurm/…), "-n | --ntasks= states the maximum number of parallel tasks lauched by the job. By default it corresponds to allocated CPU number. If this option is used with srun, then the task will be repeated n times." I would call it bad design, but that is how it is. Commented Aug 2 at 16:51
9

Tasks are processes that a job executes in parallel in one or more nodes. sbatch allocates resources for your job, but even if you request resources for multiple tasks, it will launch your job script in a single process in a single node only. srun is used to launch job steps from the batch script. --ntasks=N instructs srun to execute N copies of the job step.

For example,

#SBATCH --ntasks=2
#SBATCH --cpus-per-task=2

means that you want to run two processes in parallel, and have each process access two CPUs. sbatch will allocate four CPUs for your job and then start the batch script in a single process. Within your batch script, you can create a parallel job step using

srun --ntasks=2 --cpus-per-task=2 step.sh

This will run two processes in parallel, both of them executing the step.sh script. From the same job, you could also run

srun --ntasks=1 --cpus-per-task=4 step.sh

This would launch a single process that can access all the four GPUs (although it would issue a warning).

It's worth noting that within the allocated resources, your job script is free to do anything, and it doesn't have to use srun to create job steps (but you need srun to launch a job step in multiple nodes). For example, the following script will run both steps in parallel:

#!/bin/bash
#SBATCH --ntasks=1
step1.sh & 
step2.sh &
wait

If you want to launch job steps using srun and have two different steps run in parallel, then your job needs to allocate two tasks, and your job steps need to request only one task. You also need to provide the --exclusive argument to srun, for the job steps to use separate resources.

#!/bin/bash
#SBATCH --ntasks=2
srun --ntasks=1 --exclusive step1.sh & 
srun --ntasks=1 --exclusive step2.sh &
wait
2
  • Where is this information given in the doc ? @SeppoEnarvi Commented Jun 1, 2023 at 20:54
  • 3
    @RémyHosseinkhanBoucher I cannot recommend a good source of information. I've only ever seen the man pages that are in my opinion not so good at explaining these concepts. I've used SLURM for a decade and learned some kind of a logic behind the commands. Commented Jun 5, 2023 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.