Skip to main content
Filter by
Sorted by
Tagged with
2 votes
1 answer
85 views

Cuda with multiple GPUs: host and device report different device numbers

I am trying to utilize multiple GPUs on a HPC cluster for a simulation, but during my testing I found that it appears to run on a single GPU (device from kernel is ‘0’) although the host reports ...
Sooner's user avatar
  • 129
0 votes
0 answers
21 views

send/recv block in CuPy

I want to send a cupy array from one node to the other. The sender has the following code: import cupy import cupyx.distributed import torch.multiprocessing as mp def send(): cupy.cuda.Device(0)....
edhu's user avatar
  • 451
0 votes
0 answers
8 views

tensorflow multi-gpu problem with MirroredStrategy (UnBatchDataset?)

Thank you for your interest in the question. (Please excuse any awkward phrasing, as I am using a translation.) My main question is how to use multi-gpu for custom dataset in tensorflow. I using ...
Lee na-hyeon's user avatar
0 votes
0 answers
42 views

DDP on 8 GPUs Freezes During Training with Transformer Model—Need Help Diagnosing the Issue

I have a basic machine translation transformer model that worked well on a single GPU. However, when I tried running it on an 8-GPU setup using DDP, I initially encountered many crashes due to data ...
dsb's user avatar
  • 2,477
0 votes
0 answers
22 views

HuggingFace accelerate device error when running evaluation

I am running some experiments on a multi-GPU cluster, and I'm using accelerate. I'm trying to calculate some metrics after every batch iteration in the training dataloader. While the training code ...
M. Koopmans's user avatar
1 vote
0 answers
102 views

vllm infinite wating error in for loop (multi-gpu)

I was wondering why vllm has infinite waiting error in for loop. I made attempt_api_call() function which uses vllm to generate answer by given message. I set up vllm like this: (...
Yezun Chung's user avatar
0 votes
0 answers
27 views

K-folds cross validation in multi-GPU parallelization setup using Pytorch DDP

In a deep learning project which I am using use k-fold cross validation. I want to parallelize it to use multiple GPUs using pytorch's DDP strategy. I could not find a resource already doing this. I'm ...
Mostafa Cham's user avatar
0 votes
0 answers
20 views

Slurm mutli-node uses one node

When setting the value of n (ntasks-per-node) to any value aside from 1, it assigns tasks to only one node and ignores the other nodes. Any idea why? #SBATCH --job-name=pytorch-multinode #SBATCH --...
Mohamed Zayton's user avatar
1 vote
0 answers
64 views

How to send a buffers between 2 GPU's? [closed]

I trying to allocate buffer A and buffer B to GPU 0 and then allocate them (buffer A and buffer B) to GPU 1 and run them the a kernel that adds A + B and allocates it to buffer D (that I allocated ...
Legopart -'s user avatar
0 votes
0 answers
54 views

Loss function showing 'nan' when trying to run across multiple GPUs in Pytorch lightning

I'm trying to implement Pytorch lightning into a model I have written to solve the Allen-cahn diffusion equation. When I run the code on one gpu, the program runs as expected, but when I try to use ...
Aaronthetechguy's user avatar
0 votes
2 answers
80 views

Unable to run code on Multiple GPUs in PyTorch - Usage shows only 1 GPU is being utilized

I am training a Transformer Encoder-Decoder based model for Text summarization. The code works without any errors but uses only 1 GPU when checked with nvidia-smi. However, I want to run it on all the ...
Abid Meraj's user avatar
0 votes
0 answers
34 views

How to know how much GPU ram is occupied by a Pytorch model and how much by the data batch?

I have been experimenting with PyTorch by training different models. During runtime, I am unsure about the maximum size of my model or batch size before encountering the OOM (out of memory) error on ...
Vishruit Kulshreshtha's user avatar
0 votes
0 answers
122 views

Tensorflow GPU dependencies installation

I have been learning neural network architecture and wanted to train some on my own laptop (windows 10), I have 2 gpus, (NVIDIA Geforce MX350[driver version: 31.0.15.4680], Intel(r) Iris(r) xe ...
Mel7's user avatar
  • 19
0 votes
0 answers
107 views

Optuna HPO & Lightning Multi-GPU Training using DDP on SLURM - ValueError: World Size does not Match

I'm trying to utilize all the computational resources on a SLURM cluster to speed up my hyperparameter optimization using Optuna and Pytorch Lightning. My code works fine solely with the Pytorch ...
Zenia's user avatar
  • 13
1 vote
0 answers
49 views

Evaluation Speed is too low, and takes alot of time using HF trainer

I'm training a huge self-supervised model, when I tried to train the complete dataset, it threw cuda oom errors, to fix that I decreased batch size and added gradiant accumulation along with eval ...
Shreyas S's user avatar
1 vote
1 answer
336 views

why is FSDP not sharding my model between GPUs?

I have a model that is too large even if the batch size is 1 to fit on a single GPU. So I looked into it and it seems that FSDP is the proper way to handle this. Below, I have some code where I ...
Sam Moldenha's user avatar
2 votes
1 answer
72 views

Deploy GPU Based Application using Load Balancer with autoscalling groups

In the context of AI architecture, I am looking to optimize the distribution of API requests across multiple instances to efficiently utilize GPU resources. Specifically, I am interested in ...
Manjinder Singh's user avatar
0 votes
0 answers
50 views

How to use cuSOLVER Multi-GPU to get eigenvalues with CuPy

I have a large, dense, symmetric matrix (~50000x50000) that I want to calcuate the eigenvalues of. I tried cupy.linalg.eigvalsh(), but ran out of memory on a single GPU. I know the cuSOLVER library ...
s769's user avatar
  • 3
0 votes
0 answers
70 views

Multi Node GPU Utilization with Docker for Machine Learning Platform

My home lab server has two nodes, let's call them Node A and Node B. Node A has 4 GPUs and Node B has 2 GPUs. How can I combine that computational power with Docker containers for machine learning/...
bubba's user avatar
  • 1
0 votes
1 answer
77 views

Cannot Successfully Implement Parallel Reduction for muti-CUDA GPU

I try to run the following code which would compute the dot product of two vectors, and the code can run well when the input number of GPU is 1, that is, the Omp package isn't really used, but when ...
Ting-Kai Hsu's user avatar
0 votes
0 answers
217 views

Pytorch distribute process across nodes and gpu

I have access to 2 nodes, each has 2 GPUs. I want to have 4 processes, each has a GPU. I use nccl (if this is relevant). Here is the Slurm script I tried. I tried different combinations of setup. It ...
jasmine's user avatar
  • 221
0 votes
0 answers
113 views

Same seed across different gpus in multiple workers in huggingface/pytorch

This is more of a discussion question because I have a workaround to suit my use case. My understanding was if I am training my model on multiple GPUs (say n GPUs) in a DistributedDataParallel fashion,...
princethewinner's user avatar
0 votes
0 answers
221 views

FSDP with size_based_auto_wrap_policy freezes training

I am doing multi-node and multi-GPU training. Right now, I am optimizing my multi-GPU code for faster training. torch.cuda.set_device(self.rank) model = FSDP(model, use_orig_params=True, device_id=...
CasellaJr's user avatar
  • 458
0 votes
0 answers
85 views

How to run NVSHMEM with slurm

I'm getting started with using NVSHMEM and I wanted to start from a simple example, with not much success. #include <nvshmem.h> #include <stdio.h> int main(int argc, char *argv[]) { //...
LukeTheWalker's user avatar
0 votes
0 answers
80 views

Accessing multiple GPUs on different hosts using LSF

I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts host1 with 4 GPUs and host2 with 3 GPUs. I want to use ...
Subin Pillai's user avatar
0 votes
0 answers
293 views

CUDA out of memory while using pytorch lightning on multi-gpus

I try to finetune a diffusion model on RTX3090(24GB)x4,the details of the model are below.I use pytorch-lightning version 2.1 .I set the trainer with precision=16, accelerator="auto", ...
j hu's user avatar
  • 1
1 vote
1 answer
204 views

Getting NAN in loss function when training with multi gpu setup in tensorflow

I am recently training a deep learning model with multi GPU setup(dual 3090ti) with tf.distribute.MirroredStrategy(). In loss function, I have two types of loss; One is waveform loss and another is ...
Bisnu Sarkar's user avatar
1 vote
1 answer
159 views

Weird PyTorch Multiprocessing Error Where Main Loop Is Not Defined In __main__ | Kaggle

The following PyTorch code for single-node multi-GPU training with DDP seen here: https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu.py Gives the following error ...
Alekk's user avatar
  • 124
0 votes
0 answers
156 views

Problem with torch.nn.DataParallel - data is distributed, but not the model, it seems

Expected to train model on 4 GPUs but 3/4 of the data disappears(?) somewhere Looked all over other issues but to no avail - so asking my question here. I am trying to train the model on 4 GPUs using ...
Elizaveta Lukianova's user avatar
0 votes
0 answers
65 views

Uneven Multiple GPUs usage using Tensorflow

I am training a model using tensorflow, somehow only the first two GPU were used heavily, occupying 10000 MiB and 9290 MiB, respectively. I have eight GPUs, the six other GPUs were used but only 300 ...
el psy Congroo's user avatar
0 votes
0 answers
37 views

How to interpret multi-gpu tensorflow profile run to figure out bottleneck?

I am trying to figure out why my multi-gpu training using tensorflow MirroredStrategy is not scaling for training a 20 block x 128 filters ResNet. The single GPU run is scaling 100% with no gaps and ...
danny's user avatar
  • 1,239
0 votes
0 answers
38 views

Issues with DataLoader Reinstantiation and Resource Cleanup in Optuna Trials

Description I am using Optuna for hyperparameter optimization in my project. However, I've encountered a problem where DataLoader instances seem to accumulate over multiple trials. Despite creating a ...
fdsfsd's user avatar
  • 1
1 vote
0 answers
36 views

Very strange timing in Nvidia Visual profiler

I'm trying to measure some programs, including some of those included in the CUDA samples, with nvprof in my environment with two Nvidia Quadro Q1000 GPUs. For 5_Domain_Specific/MonteCarloMultiGPU ...
Antonello Cioffi's user avatar
2 votes
0 answers
1k views

Why does my device_map="auto" in transformers.pipline uses CPU only even though GPUs are available?

I came across this problem when trying out LLaMa 2 (13B version) on a 8X32GB-GPU server. The pipeline setting is like below: pipeline = transformers.pipeline( "text-generation", ...
Phil-Antony's user avatar
0 votes
1 answer
65 views

Moving tensors between GPUs in Lightning

How can I move tensors from one gpu to another in training_step of pl.LightningModule? In torch-based pipeline I use the following function to move tensors during multi-gpu fitting procedure: def ...
The Hidden Reverse's user avatar
1 vote
2 answers
364 views

cuDNN Error: CUDNN_STATUS_BAD_PARAM. Can someone tell what does this mean, why this is occurring?

I am trying to train an object detection model with yolov4 using multiple GPUs (tesla T4), but between around 1000 or 2000 iterations it's giving the below error. (next mAP calculation at 1214 ...
Malla Raraju's user avatar
0 votes
0 answers
29 views

Google Compute Engine Tesla K80 correct image

I am using Google compute Engine to do some ML. I want 8 GPUs and the cheapest is the Tesla K80. I have tried many combinations of image and found every time that the GPUs are Unclaimed and so ...
mathlete42's user avatar
2 votes
0 answers
749 views

What is the proper way to do multi-GPU inference with Pytorch?

I'm facing some issues with multi-GPU inference using pytorch and pytorch-lightning models. At inference time, I need to use two different models in an auto-regressive manner. Since the auto-...
erik's user avatar
  • 21
0 votes
0 answers
179 views

TypeError: cannot pickle '_thread.lock' object when wrapping Pytorch Distributed Data Parallel in a class

Here is a minimal code: import threading import torch class DistributedMission: def __init__(self): self.lock = threading.Lock() self.world_size = 8 def start(self): ...
landings's user avatar
  • 730
0 votes
0 answers
70 views

Gathering result using Distributed Data Parallel needs too much video memory

the following is my task method used for multi-GPU predicting. def worker(self, rank:int, pickle_file_name:str): torch.distributed.init_process_group('nccl', rank=rank, world_size=self.world_size) ...
landings's user avatar
  • 730
0 votes
0 answers
192 views

My Pytorch multi-GPU prediction code freezes at `spawn()`

The following small code does multi-GPU prediction using Pytorch. I launch multiple tasks using torch.multiprocessing.spawn(). Inside task, I put no real prediction code. Just putting something into a ...
landings's user avatar
  • 730
0 votes
0 answers
241 views

How can I train a yolov8 model with multi-gpu and rect=True?

I need to train a YOLOv8 model to detect objects in some extremely flat and elongated images. But I find that 'rect' mode is incompatible with multiple GPUs, it warns me that 'rect' mode is ...
MsD日月's user avatar
1 vote
0 answers
125 views

Keras multi-GPU training, GPU resource can't be shared, causing training failure

I tried to apply keras multi-GPU to train my model with the following code: tf.keras.backend.clear_session() strategy = tf.distribute.MirroredStrategy() print("Number of devices: {}".format(...
chengzhangpei's user avatar
0 votes
0 answers
106 views

Communication using PyTorch distributed package extremely slow

I am using pytorch distributed package to write a simple program that let multiple GPUs on a single server communicate with each other. Here is the code: def run(rank, world_size): # torch.cuda....
DuoLife-QNL's user avatar
0 votes
0 answers
8 views

Interference between Pyro4 and DDP process communication

I'm running a distributed machine learning training loop inside the remote function. The distributed training code is using pytorch-lightning and communication between the calling function and remote ...
Dipti Sengupta's user avatar
0 votes
2 answers
1k views

YoloV7 - Multi-GPU constantly gives RunTime Error

I am using YoloV7 to run a training session for custom object detection. My environment is as follows: OS: Ubuntu 22.04 Python : 3.10 Torch Version : '2.1.0+cu121' I am using AWS EC2 - g5.2xlarge ...
Apricot's user avatar
  • 3,001
1 vote
0 answers
275 views

How to train a model on a cluster with multiple GPUs (single node)

I wish to train a model on a multiple-GPUs, 1-Node cluster on AML (Azure ML). I have my cluster (STANDARD_ND40RS_V2, which has 8 V100) I have my Dataset, my Environment (docker), and my script. When ...
gozi's user avatar
  • 95
0 votes
0 answers
66 views

Problem with running a tensorflow-based script on multiple GPUs of a cluster

I want to train a 3D model based on tensorflow. I'm using a node in a cluster with 8 GPUs (Geforce GTX 1080 ti, with 1045 MB memory for each one). My inputs are 3D CT images. When I reduce the image ...
Mehdi MA's user avatar
0 votes
1 answer
524 views

How to clear GPU memory after training each fold

I am getting an error after a few folds of training is done.. The error states: Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.15GiB (rounded to 17338417152)requested by op _EagerConst ...
Subhram Dasgupta's user avatar
1 vote
0 answers
147 views

torch DDP Multi-GPU gives low accuracy metric

I am trying Multi-GPU, single machine DDP training in PyTorch (CIFAR-10 + ResNet-18 setup). You can refer to the model architecture code here and the full training code here. Within main() function, ...
Arun's user avatar
  • 2,446

1
2 3 4 5
9