Questions tagged [slurm]
Slurm (formerly spelled SLURM) is an open-source resource manager designed for Linux HPC clusters of all sizes.
1,792
questions
0
votes
0
answers
8
views
Waiting completion of running jobs before proceeding to next script [duplicate]
I have written a main shell script to do RNA-Seq data analysis for multiple samples. After a few initial commands, it has a command that initiates further sbatch jobs (one for each sample) using ...
0
votes
1
answer
41
views
SLURM: Run Same Parallel Script on Multiple Files
What I'm trying to do is to run the same python script on thousands of input csv files. The python script is a single-process script that takes as input a single file and creates an output file with ...
0
votes
0
answers
13
views
SLURM - DB and MaxArray MaxJob
I changed the SLURM array configuration in
MaxArraySize=100001
MaxJobCount=1000000
but can SLURM (db RAM ???) manage all those jobs/arrays, I mean, should I change any other thing in the conf file ...
-3
votes
0
answers
102
views
Running python code on Research Computing Cluster [closed]
In our university, we have a big cluster but I am not able to understand how to use it ( how to create a conda environment and then run the python code).
Below is a sample script of run.sh file to run ...
0
votes
1
answer
38
views
How to pass two variables in bash to a python script
I am trying to pass an array index and filename to a python script bu tI just can't get it to work:
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --output=Test.txt
#SBATCH --array=0-1
for Folder in ~/...
1
vote
1
answer
20
views
Sbatch resouce allocation for mult-threaded application
After reading the very informative post, HPC cluster: select the number of CPUs and threads in SLURM sbatch, I understand the difference b/w the --ntasks, --cpus-per-task and --ntasks-per-node flags ...
2
votes
1
answer
23
views
Running a job on CPU by default, but on GPU when available in Slurm
Is there a way to submit a job to Slurm with sbatch and use the gpu if available, but run on cpu if there is no gpu available?
Setting: #SBATCH --gres=gpu:1 only runs on nodes where a gpu is available....
0
votes
0
answers
48
views
+50
Slurm cgroups Plugin Errors on Ubuntu Nodes
We're pretty new to using Slurm and have run into some tricky issues with the cgroups plugin. We started diving into Slurm to get better at managing our computing resources, especially for the more ...
0
votes
0
answers
30
views
Are there benefits to using DDP with pandarallel?
I have a ML task, and as a preprocessing step, there is a lot of work on CPU to be done which can take >1 hour. Before, I was using the pandarallel library to parallelize this work, the ...
1
vote
0
answers
28
views
How to set CPU settings properly for slurm job?
I have a job that uses a moderate amount of the allocated memory, and all of the CPU. How should I adjust the CPU settings, memory settings, or both? The job info, and its slurm settings are below:
...
0
votes
1
answer
31
views
Confusing sacct Job Accounting Fields: `TresUsage[In|Out][Max|Tot]`
We currently run
$ sacct --version
slurm 21.08.6
and I am utterly confused about the various Job Accounting fiels.
Looking at man sacct under Job Accounting Fields we see numerous TRES fields, but I ...
0
votes
0
answers
36
views
wandb error when running as a subprocess: ManagerConnectionRefusedError: Connection to wandb service failed since the process is not available
I’m having a problem when I try to run a subprocess (with Popen) in my python script that executes a bash command (slurm sbatch) on a different computing node.
The error happens during wandb.init():
...
0
votes
0
answers
18
views
problem on slurm setting: network connection refused
I successfully completed setup slurm on 2 nodes.
And scontrol show nodes, sinfo -a, srun -w <test_node> python test.py gave me correct results.
I tried to modify some configuration related to ...
0
votes
2
answers
37
views
Snakemake, slurm and memory
I am struggling to understand how snakemake submits jobs to slurm.
When I have a basic slurm sbatch script I usually add a line, such as
#SBATCH --mem=5G
to determine that slurm may use 5 gigabytes (...
1
vote
1
answer
27
views
Is redirecting output with > redundant when using slurm?
I am running my program with slurm. I want all output (and all errors) to go to logfile.log. My sbatch script is
#!/bin/bash
#SBATCH --output=logfile.log
#SBATCH --error=logfile.log
srun PROGRAM
I ...
0
votes
0
answers
30
views
SLURM job use IP of the node
I have 3 workers/nodes and each worker is connected to internet (worker1 ip: x.x.x.101, worker2 ip: x.x.x.102, worker3 ip: x.x.x.103)
How can I say to the job, running on worker3, to use the worker3 ...
0
votes
1
answer
32
views
Reserve some CPUs on a node for GPU jobs
I'm setting up a GPU cluster with Slurm. The nodes have a variable number of CPU cores (8-32) and a variable number of GPUs (1-4). The GPU jobs that are going to run generally require very little CPU ...
1
vote
1
answer
34
views
SLURM/NFS cluster shared filesystem locked permissions on some users
Background
I used slurm to set up a 3-node cluster, one control node and 3 compute nodes (the control node doubles as a compute node). All of the nodes are on linux mint and they all communicate ...
0
votes
1
answer
90
views
Use FP exception traps (-ffpe-trap/-fpe0) for code linked against SIGFPE-unsafe library (libxml2)
I have Fortran code, for which I would like to enable trapping of floating-point exceptions (as recommended by the compiler man pages). But when I do so, the binary will no longer run in a Slurm queue....
2
votes
1
answer
48
views
SLURM array Add variable in #SBATCH options
I have a SLURM array
#SBATCH --array=0-1000%10
and I need to add a variable instead of "1000"
to be something like
#! /bin/bash
num=$1
#SBATCH --array=1-${num}%10
or if there is other way ...
0
votes
0
answers
22
views
Slurm Error: Error in saveRDS(fit, paste("outputs3/fit_", mod, ".rds", sep = "")) : error writing to connection
I have a batch script that I am trying to run. I've made some changes to the original script (which ran and saved successfully) so I wanted to rerun the models and save to a different output folder. I ...
1
vote
1
answer
38
views
How to hold jobs for a user so that a total cpu number isn't reached in slurm?
I am submitting a bunch of array jobs 4 sets of 5 with 8 cpus each, so 4x5x8=160 cpus total. I would like to keep the running amount to below 100>2x5x8+1x2x8 cpus total, because I need to let ...
0
votes
0
answers
20
views
Submit tasks that run in parallel over multiple CPUs
Running my code does not require much time. When I launch multiple runs on my laptop, I use joblib and it works nicely. I now want to launch my runs on a Slurm-based cluster. I can't understand what ...
0
votes
0
answers
24
views
Slurm Can not set different Gpu Type on same node
I'm trying to add a graphics card to the slurm node. Currently, I have two p104s and one 1080ti. I have a problem setting up slurm.conf and gres.conf.
I want to assign a type to each graphics card
...
1
vote
0
answers
23
views
Slurm, run job only when sufficient memory is available
How can i run a job on slurm using srun such that it goes into queue when the neccessary resources such as memory per gpu is not available, and only runs when it is.
I tried using the flag mem-per-gpu,...
0
votes
0
answers
19
views
how start_time is calculated for slurm multi-cluster
I'm having a hard time figuring out the distribution of jobs between 2 clusters in a Slurm multi-cluster environment. The documentation says that each job is submitted to the cluster that provides the ...
0
votes
0
answers
62
views
torch.distributed.get_world_size() and torch.cuda.device_count() returning different numbers, getting invalid device ordinal error
I'm trying to used tensor parallelization across multiple GPUs in pytorch, specifically 2 Nvidia A100s, to spread a model which is too big for one GPU across multiple using a slurm-based HPC. I'm ...
0
votes
0
answers
28
views
Attaching Debugger to Slurm compute node using Debugpy
I'm trying to debug code running on a SLURM compute node. I'm using VS Code to connect to the head node where I develop my code. To debug I have first to allocate a node to get access to a GPU. I ...
0
votes
1
answer
30
views
Finding SLURM node running PSQL server
I have the following problem:
I can connect via SSH to a SLURM cluster running Red Hat Enterprise Linux 7, however every time I connect I get assigned a different login node. From any of the login ...
0
votes
1
answer
26
views
How to send a job to nodes of different types in SLURM
I work at a HPC centre where we use SLURM to manage queues and I'm looking for a way to force a job divided by tasks to be sent to nodes of different types. For example, let's say we have two types of ...
0
votes
0
answers
26
views
Running multiple bash scripts using sbatch [duplicate]
I an using the cluster, and I have been running my bash scripts using
$sbatch file1.sh
$sbatch file2.sh
Is there a way that I can run all my bash scripts at once, maybe using a wildcard?
0
votes
2
answers
22
views
'--dependendy=aftercorr:Submitted' slurm option unrecognized
I have a script to run in slurm. The script works with job-array aftercorr, or at least that's what I want it to do. Basically it runs the first Job-array without any issue, however, it does not ...
0
votes
1
answer
68
views
How to run multiple jobs, one per GPU, with SLURM?
Apologies if this has be asked/answered before, but even after reading everything I can find, I am struggling to get SLURM to do what I want.
Let's say I have a machine with 4 GPUs. I want to train 4 ...
1
vote
1
answer
21
views
SLURM reported memory consumption
I'm trying to understand how much memory a simple job consumes (based on this python script). I run the same script on a single worker with different numbers of cores allocated.
The results of this ...
3
votes
2
answers
89
views
Running parallel multiprocessing python scripts with slurm
I'm trying to run simultaneously using slurm a python script that internally parallelizes a process using multiprocessing. The goal of the python script is to solve a stochastic differential equation ...
0
votes
0
answers
24
views
How to get reliable results on SLURM for performance tests?
I want to benchmark two algorithms ALG1 and ALG2 against each other using SLURM. Both binaries ALG1.o and ALG2.o contain serial implementations of the algorithms. In now, that ALG1.o is always faster ...
0
votes
0
answers
12
views
Nginx load balancing against Slurm API
What strategy of load balancing in k8s would be recommended between Slurm (think SGE) clusters?
The reason I raise this is Slurm has a Pythonic API governing the queuing system.
Information can freely ...
0
votes
1
answer
47
views
Is it possible to run multiple Slurm job-steps which us a single GPU?
I have been trying to use Slurm to run multiple job steps on a single GPU. I know memory won't be an issue if the jobs were able to run, however only the first job step will run in the below sequence.
...
0
votes
0
answers
24
views
What does a Slurm job have to do with ssh?
I am trying to run my first Slurm job. It is hanging right after the executable begins running. I get the error
Could not create directory '/p/home/mydir/.ssh'.
Could not create directory '/p/home/...
1
vote
1
answer
19
views
SLURM/sbatch: How to pass both a pointer to my data and the SLURM job ID to a job script
I am using a job array to process a large number of files, and I am able to pass a pointer from my array to the specific data file to be processed in the job script, but I want to also pass the ...
0
votes
0
answers
47
views
Why am I getting duplicate(?) outputs for my slurm jobs
I've started working with Slurm, so I've written a simple script to try it out.
Basically, it intends to submit 10 jobs as a job array, and each job just echos its index (0, ..., 9).
But I'm a bit ...
0
votes
0
answers
23
views
Difference between `salloc srun` and `srun` running on login node
I don't see the difference. Can you explain the difference and maybe give some usecase for both of the usages?
1
vote
1
answer
39
views
Scheduling one task per gpu on multiple nodes
I am trying to schedule 8 tasks on two separate nodes.
Each node has 4 GPUs and I want to assign one per tasks.
The following code works fine on 1 node and 4 GPUS:
#!/bin/bash
#
#SBATCH --time 24:00:...
0
votes
0
answers
15
views
How to register a custom container or invoker to the OpenWhisk controller for invocation
I want to deploy OpenWhisk on an HPC cluster to build a serverless platform. However, I hope to use Slurm to manage the actual resource allocation.
The problem I am running into now is how to start an ...
0
votes
0
answers
74
views
Submitting Slurm job to head node via compute node?
I've set up a Slurm cluster on AWS ParallelCluster for a customer who needs to be able to launch nested Slurm jobs. For example, from a login node, we need to be able to launch a single job on a ...
0
votes
1
answer
35
views
Slurm Batch Script Save to Variable or Append
I would like to be able to append my slurm batch scripts to my error file. I submit these on the HPC using sbatch -J "my_run_name" runscript.slurm. The content of runscript.slurm is ...
0
votes
0
answers
17
views
unpredictable behavior of slurm multi-cluster
I have 2 slurm clusters: cluster A with 3 compute nodes, each node has 32 CPUs; Cluster B with 4 compute nodes, each node has 8 CPUs. I'm using slurm multicluster on clusters A and B. I tried to run ...
1
vote
1
answer
54
views
Slurm : srun with multiple executables
Is there a way to have run commands run one after the other under a single srun? Additionally, can this be done without srunning a seperate file that has all the commands ?
Normally we would do :
srun ...
0
votes
1
answer
38
views
Make sure there's always N jobs running/allocated out of a batch of M with SLURM
I need to use SLURM to perform cluster computing and I would like to reach the following outcome: out of a batch file containing M tasks, only N are in execution/reserved at any one time.
I have a ...
0
votes
1
answer
46
views
Can I create a job name that reflects the array task ID?
I am using a high-performance research cloud at my university that utilizes the Slurm workload manager (I am very unfamiliar with this type of computing). I have figured out enough to get a batch job ...