Questions tagged [slurm]

Slurm (formerly spelled SLURM) is an open-source resource manager designed for Linux HPC clusters of all sizes.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Waiting completion of running jobs before proceeding to next script [duplicate]

I have written a main shell script to do RNA-Seq data analysis for multiple samples. After a few initial commands, it has a command that initiates further sbatch jobs (one for each sample) using ...
Muhammad Sajid Tahir's user avatar
0 votes
1 answer
41 views

SLURM: Run Same Parallel Script on Multiple Files

What I'm trying to do is to run the same python script on thousands of input csv files. The python script is a single-process script that takes as input a single file and creates an output file with ...
camilo's user avatar
  • 3
0 votes
0 answers
13 views

SLURM - DB and MaxArray MaxJob

I changed the SLURM array configuration in MaxArraySize=100001 MaxJobCount=1000000 but can SLURM (db RAM ???) manage all those jobs/arrays, I mean, should I change any other thing in the conf file ...
Emma Athan's user avatar
-3 votes
0 answers
102 views

Running python code on Research Computing Cluster [closed]

In our university, we have a big cluster but I am not able to understand how to use it ( how to create a conda environment and then run the python code). Below is a sample script of run.sh file to run ...
mchaudh4's user avatar
0 votes
1 answer
38 views

How to pass two variables in bash to a python script

I am trying to pass an array index and filename to a python script bu tI just can't get it to work: #!/bin/bash #SBATCH --job-name=test #SBATCH --output=Test.txt #SBATCH --array=0-1 for Folder in ~/...
nkehoe's user avatar
  • 33
1 vote
1 answer
20 views

Sbatch resouce allocation for mult-threaded application

After reading the very informative post, HPC cluster: select the number of CPUs and threads in SLURM sbatch, I understand the difference b/w the --ntasks, --cpus-per-task and --ntasks-per-node flags ...
DOOM's user avatar
  • 1,202
2 votes
1 answer
23 views

Running a job on CPU by default, but on GPU when available in Slurm

Is there a way to submit a job to Slurm with sbatch and use the gpu if available, but run on cpu if there is no gpu available? Setting: #SBATCH --gres=gpu:1 only runs on nodes where a gpu is available....
Curantil's user avatar
0 votes
0 answers
48 views
+50

Slurm cgroups Plugin Errors on Ubuntu Nodes

We're pretty new to using Slurm and have run into some tricky issues with the cgroups plugin. We started diving into Slurm to get better at managing our computing resources, especially for the more ...
Francisco Maria Calisto's user avatar
0 votes
0 answers
30 views

Are there benefits to using DDP with pandarallel?

I have a ML task, and as a preprocessing step, there is a lot of work on CPU to be done which can take >1 hour. Before, I was using the pandarallel library to parallelize this work, the ...
Alex Li's user avatar
  • 56
1 vote
0 answers
28 views

How to set CPU settings properly for slurm job?

I have a job that uses a moderate amount of the allocated memory, and all of the CPU. How should I adjust the CPU settings, memory settings, or both? The job info, and its slurm settings are below: ...
somewhere's user avatar
0 votes
1 answer
31 views

Confusing sacct Job Accounting Fields: `TresUsage[In|Out][Max|Tot]`

We currently run $ sacct --version slurm 21.08.6 and I am utterly confused about the various Job Accounting fiels. Looking at man sacct under Job Accounting Fields we see numerous TRES fields, but I ...
Stefan's user avatar
  • 1,727
0 votes
0 answers
36 views

wandb error when running as a subprocess: ManagerConnectionRefusedError: Connection to wandb service failed since the process is not available

I’m having a problem when I try to run a subprocess (with Popen) in my python script that executes a bash command (slurm sbatch) on a different computing node. The error happens during wandb.init(): ...
schlaepf's user avatar
0 votes
0 answers
18 views

problem on slurm setting: network connection refused

I successfully completed setup slurm on 2 nodes. And scontrol show nodes, sinfo -a, srun -w <test_node> python test.py gave me correct results. I tried to modify some configuration related to ...
MyungHa Kwon's user avatar
0 votes
2 answers
37 views

Snakemake, slurm and memory

I am struggling to understand how snakemake submits jobs to slurm. When I have a basic slurm sbatch script I usually add a line, such as #SBATCH --mem=5G to determine that slurm may use 5 gigabytes (...
Laura's user avatar
  • 3
1 vote
1 answer
27 views

Is redirecting output with > redundant when using slurm?

I am running my program with slurm. I want all output (and all errors) to go to logfile.log. My sbatch script is #!/bin/bash #SBATCH --output=logfile.log #SBATCH --error=logfile.log srun PROGRAM I ...
fxm's user avatar
  • 135
0 votes
0 answers
30 views

SLURM job use IP of the node

I have 3 workers/nodes and each worker is connected to internet (worker1 ip: x.x.x.101, worker2 ip: x.x.x.102, worker3 ip: x.x.x.103) How can I say to the job, running on worker3, to use the worker3 ...
Emma Athan's user avatar
0 votes
1 answer
32 views

Reserve some CPUs on a node for GPU jobs

I'm setting up a GPU cluster with Slurm. The nodes have a variable number of CPU cores (8-32) and a variable number of GPUs (1-4). The GPU jobs that are going to run generally require very little CPU ...
Atomjack's user avatar
1 vote
1 answer
34 views

SLURM/NFS cluster shared filesystem locked permissions on some users

Background I used slurm to set up a 3-node cluster, one control node and 3 compute nodes (the control node doubles as a compute node). All of the nodes are on linux mint and they all communicate ...
floating_protein's user avatar
0 votes
1 answer
90 views

Use FP exception traps (-ffpe-trap/-fpe0) for code linked against SIGFPE-unsafe library (libxml2)

I have Fortran code, for which I would like to enable trapping of floating-point exceptions (as recommended by the compiler man pages). But when I do so, the binary will no longer run in a Slurm queue....
H. Weirauch's user avatar
2 votes
1 answer
48 views

SLURM array Add variable in #SBATCH options

I have a SLURM array #SBATCH --array=0-1000%10 and I need to add a variable instead of "1000" to be something like #! /bin/bash num=$1 #SBATCH --array=1-${num}%10 or if there is other way ...
Emma Athan's user avatar
0 votes
0 answers
22 views

Slurm Error: Error in saveRDS(fit, paste("outputs3/fit_", mod, ".rds", sep = "")) : error writing to connection

I have a batch script that I am trying to run. I've made some changes to the original script (which ran and saved successfully) so I wanted to rerun the models and save to a different output folder. I ...
megsruppUNBC's user avatar
1 vote
1 answer
38 views

How to hold jobs for a user so that a total cpu number isn't reached in slurm?

I am submitting a bunch of array jobs 4 sets of 5 with 8 cpus each, so 4x5x8=160 cpus total. I would like to keep the running amount to below 100>2x5x8+1x2x8 cpus total, because I need to let ...
Jason M Gray's user avatar
0 votes
0 answers
20 views

Submit tasks that run in parallel over multiple CPUs

Running my code does not require much time. When I launch multiple runs on my laptop, I use joblib and it works nicely. I now want to launch my runs on a Slurm-based cluster. I can't understand what ...
Simon's user avatar
  • 5,158
0 votes
0 answers
24 views

Slurm Can not set different Gpu Type on same node

I'm trying to add a graphics card to the slurm node. Currently, I have two p104s and one 1080ti. I have a problem setting up slurm.conf and gres.conf. I want to assign a type to each graphics card ...
soulreaper's user avatar
1 vote
0 answers
23 views

Slurm, run job only when sufficient memory is available

How can i run a job on slurm using srun such that it goes into queue when the neccessary resources such as memory per gpu is not available, and only runs when it is. I tried using the flag mem-per-gpu,...
Wj210's user avatar
  • 19
0 votes
0 answers
19 views

how start_time is calculated for slurm multi-cluster

I'm having a hard time figuring out the distribution of jobs between 2 clusters in a Slurm multi-cluster environment. The documentation says that each job is submitted to the cluster that provides the ...
shambakey1's user avatar
0 votes
0 answers
62 views

torch.distributed.get_world_size() and torch.cuda.device_count() returning different numbers, getting invalid device ordinal error

I'm trying to used tensor parallelization across multiple GPUs in pytorch, specifically 2 Nvidia A100s, to spread a model which is too big for one GPU across multiple using a slurm-based HPC. I'm ...
DLS's user avatar
  • 15
0 votes
0 answers
28 views

Attaching Debugger to Slurm compute node using Debugpy

I'm trying to debug code running on a SLURM compute node. I'm using VS Code to connect to the head node where I develop my code. To debug I have first to allocate a node to get access to a GPU. I ...
Nicolai's user avatar
  • 62
0 votes
1 answer
30 views

Finding SLURM node running PSQL server

I have the following problem: I can connect via SSH to a SLURM cluster running Red Hat Enterprise Linux 7, however every time I connect I get assigned a different login node. From any of the login ...
Liuka's user avatar
  • 299
0 votes
1 answer
26 views

How to send a job to nodes of different types in SLURM

I work at a HPC centre where we use SLURM to manage queues and I'm looking for a way to force a job divided by tasks to be sent to nodes of different types. For example, let's say we have two types of ...
Willy's user avatar
  • 1
0 votes
0 answers
26 views

Running multiple bash scripts using sbatch [duplicate]

I an using the cluster, and I have been running my bash scripts using $sbatch file1.sh $sbatch file2.sh Is there a way that I can run all my bash scripts at once, maybe using a wildcard?
thole's user avatar
  • 117
0 votes
2 answers
22 views

'--dependendy=aftercorr:Submitted' slurm option unrecognized

I have a script to run in slurm. The script works with job-array aftercorr, or at least that's what I want it to do. Basically it runs the first Job-array without any issue, however, it does not ...
Cynthia's user avatar
0 votes
1 answer
68 views

How to run multiple jobs, one per GPU, with SLURM?

Apologies if this has be asked/answered before, but even after reading everything I can find, I am struggling to get SLURM to do what I want. Let's say I have a machine with 4 GPUs. I want to train 4 ...
Vivek's user avatar
  • 497
1 vote
1 answer
21 views

SLURM reported memory consumption

I'm trying to understand how much memory a simple job consumes (based on this python script). I run the same script on a single worker with different numbers of cores allocated. The results of this ...
Raphael's user avatar
  • 188
3 votes
2 answers
89 views

Running parallel multiprocessing python scripts with slurm

I'm trying to run simultaneously using slurm a python script that internally parallelizes a process using multiprocessing. The goal of the python script is to solve a stochastic differential equation ...
dsqubitdo's user avatar
0 votes
0 answers
24 views

How to get reliable results on SLURM for performance tests?

I want to benchmark two algorithms ALG1 and ALG2 against each other using SLURM. Both binaries ALG1.o and ALG2.o contain serial implementations of the algorithms. In now, that ALG1.o is always faster ...
ahul-fell-awen's user avatar
0 votes
0 answers
12 views

Nginx load balancing against Slurm API

What strategy of load balancing in k8s would be recommended between Slurm (think SGE) clusters? The reason I raise this is Slurm has a Pythonic API governing the queuing system. Information can freely ...
M__'s user avatar
  • 624
0 votes
1 answer
47 views

Is it possible to run multiple Slurm job-steps which us a single GPU?

I have been trying to use Slurm to run multiple job steps on a single GPU. I know memory won't be an issue if the jobs were able to run, however only the first job step will run in the below sequence. ...
JBartlett_6's user avatar
0 votes
0 answers
24 views

What does a Slurm job have to do with ssh?

I am trying to run my first Slurm job. It is hanging right after the executable begins running. I get the error Could not create directory '/p/home/mydir/.ssh'. Could not create directory '/p/home/...
bob.sacamento's user avatar
1 vote
1 answer
19 views

SLURM/sbatch: How to pass both a pointer to my data and the SLURM job ID to a job script

I am using a job array to process a large number of files, and I am able to pass a pointer from my array to the specific data file to be processed in the job script, but I want to also pass the ...
Char19's user avatar
  • 13
0 votes
0 answers
47 views

Why am I getting duplicate(?) outputs for my slurm jobs

I've started working with Slurm, so I've written a simple script to try it out. Basically, it intends to submit 10 jobs as a job array, and each job just echos its index (0, ..., 9). But I'm a bit ...
ljg's user avatar
  • 1
0 votes
0 answers
23 views

Difference between `salloc srun` and `srun` running on login node

I don't see the difference. Can you explain the difference and maybe give some usecase for both of the usages?
Yuxuan Lu's user avatar
  • 336
1 vote
1 answer
39 views

Scheduling one task per gpu on multiple nodes

I am trying to schedule 8 tasks on two separate nodes. Each node has 4 GPUs and I want to assign one per tasks. The following code works fine on 1 node and 4 GPUS: #!/bin/bash # #SBATCH --time 24:00:...
Gianluca De Stefano's user avatar
0 votes
0 answers
15 views

How to register a custom container or invoker to the OpenWhisk controller for invocation

I want to deploy OpenWhisk on an HPC cluster to build a serverless platform. However, I hope to use Slurm to manage the actual resource allocation. The problem I am running into now is how to start an ...
hfldqwe's user avatar
  • 29
0 votes
0 answers
74 views

Submitting Slurm job to head node via compute node?

I've set up a Slurm cluster on AWS ParallelCluster for a customer who needs to be able to launch nested Slurm jobs. For example, from a login node, we need to be able to launch a single job on a ...
Cody Oliveros's user avatar
0 votes
1 answer
35 views

Slurm Batch Script Save to Variable or Append

I would like to be able to append my slurm batch scripts to my error file. I submit these on the HPC using sbatch -J "my_run_name" runscript.slurm. The content of runscript.slurm is ...
Nukesub's user avatar
  • 193
0 votes
0 answers
17 views

unpredictable behavior of slurm multi-cluster

I have 2 slurm clusters: cluster A with 3 compute nodes, each node has 32 CPUs; Cluster B with 4 compute nodes, each node has 8 CPUs. I'm using slurm multicluster on clusters A and B. I tried to run ...
shambakey1's user avatar
1 vote
1 answer
54 views

Slurm : srun with multiple executables

Is there a way to have run commands run one after the other under a single srun? Additionally, can this be done without srunning a seperate file that has all the commands ? Normally we would do : srun ...
Liam F-A's user avatar
  • 138
0 votes
1 answer
38 views

Make sure there's always N jobs running/allocated out of a batch of M with SLURM

I need to use SLURM to perform cluster computing and I would like to reach the following outcome: out of a batch file containing M tasks, only N are in execution/reserved at any one time. I have a ...
Michele Ancis's user avatar
0 votes
1 answer
46 views

Can I create a job name that reflects the array task ID?

I am using a high-performance research cloud at my university that utilizes the Slurm workload manager (I am very unfamiliar with this type of computing). I have figured out enough to get a batch job ...
c.custer's user avatar
  • 409

1
2 3 4 5
36