413
questions
2
votes
1
answer
85
views
Cuda with multiple GPUs: host and device report different device numbers
I am trying to utilize multiple GPUs on a HPC cluster for a simulation, but during my testing I found that it appears to run on a single GPU (device from kernel is ‘0’) although the host reports ...
0
votes
0
answers
21
views
send/recv block in CuPy
I want to send a cupy array from one node to the other.
The sender has the following code:
import cupy
import cupyx.distributed
import torch.multiprocessing as mp
def send():
cupy.cuda.Device(0)....
0
votes
0
answers
8
views
tensorflow multi-gpu problem with MirroredStrategy (UnBatchDataset?)
Thank you for your interest in the question.
(Please excuse any awkward phrasing, as I am using a translation.)
My main question is how to use multi-gpu for custom dataset in tensorflow.
I using
...
0
votes
0
answers
42
views
DDP on 8 GPUs Freezes During Training with Transformer Model—Need Help Diagnosing the Issue
I have a basic machine translation transformer model that worked well on a single GPU. However, when I tried running it on an 8-GPU setup using DDP, I initially encountered many crashes due to data ...
0
votes
0
answers
22
views
HuggingFace accelerate device error when running evaluation
I am running some experiments on a multi-GPU cluster, and I'm using accelerate. I'm trying to calculate some metrics after every batch iteration in the training dataloader. While the training code ...
1
vote
0
answers
102
views
vllm infinite wating error in for loop (multi-gpu)
I was wondering why vllm has infinite waiting error in for loop.
I made attempt_api_call() function which uses vllm to generate answer by given message.
I set up vllm like this:
(...
0
votes
0
answers
27
views
K-folds cross validation in multi-GPU parallelization setup using Pytorch DDP
In a deep learning project which I am using use k-fold cross validation. I want to parallelize it to use multiple GPUs using pytorch's DDP strategy. I could not find a resource already doing this. I'm ...
0
votes
0
answers
20
views
Slurm mutli-node uses one node
When setting the value of n (ntasks-per-node) to any value aside from 1, it assigns tasks to only one node and ignores the other nodes. Any idea why?
#SBATCH --job-name=pytorch-multinode
#SBATCH --...
1
vote
0
answers
64
views
How to send a buffers between 2 GPU's? [closed]
I trying to allocate buffer A and buffer B to GPU 0 and then allocate them (buffer A and buffer B) to GPU 1 and run them the a kernel that adds A + B and allocates it to buffer D (that I allocated ...
0
votes
0
answers
54
views
Loss function showing 'nan' when trying to run across multiple GPUs in Pytorch lightning
I'm trying to implement Pytorch lightning into a model I have written to solve the Allen-cahn diffusion equation. When I run the code on one gpu, the program runs as expected, but when I try to use ...
0
votes
2
answers
80
views
Unable to run code on Multiple GPUs in PyTorch - Usage shows only 1 GPU is being utilized
I am training a Transformer Encoder-Decoder based model for Text summarization. The code works without any errors but uses only 1 GPU when checked with nvidia-smi. However, I want to run it on all the ...
0
votes
0
answers
34
views
How to know how much GPU ram is occupied by a Pytorch model and how much by the data batch?
I have been experimenting with PyTorch by training different models. During runtime, I am unsure about the maximum size of my model or batch size before encountering the OOM (out of memory) error on ...
0
votes
0
answers
122
views
Tensorflow GPU dependencies installation
I have been learning neural network architecture and wanted to train some on my own
laptop (windows 10),
I have 2 gpus, (NVIDIA Geforce MX350[driver version: 31.0.15.4680], Intel(r) Iris(r) xe ...
0
votes
0
answers
107
views
Optuna HPO & Lightning Multi-GPU Training using DDP on SLURM - ValueError: World Size does not Match
I'm trying to utilize all the computational resources on a SLURM cluster to speed up my hyperparameter optimization using Optuna and Pytorch Lightning.
My code works fine solely with the Pytorch ...
1
vote
0
answers
49
views
Evaluation Speed is too low, and takes alot of time using HF trainer
I'm training a huge self-supervised model, when I tried to train the complete dataset, it threw cuda oom errors, to fix that I decreased batch size and added gradiant accumulation along with eval ...
1
vote
1
answer
336
views
why is FSDP not sharding my model between GPUs?
I have a model that is too large even if the batch size is 1 to fit on a single GPU. So I looked into it and it seems that FSDP is the proper way to handle this. Below, I have some code where I ...
2
votes
1
answer
72
views
Deploy GPU Based Application using Load Balancer with autoscalling groups
In the context of AI architecture, I am looking to optimize the distribution of API requests across multiple instances to efficiently utilize GPU resources. Specifically, I am interested in ...
0
votes
0
answers
50
views
How to use cuSOLVER Multi-GPU to get eigenvalues with CuPy
I have a large, dense, symmetric matrix (~50000x50000) that I want to calcuate the eigenvalues of.
I tried cupy.linalg.eigvalsh(), but ran out of memory on a single GPU. I know the cuSOLVER library ...
0
votes
0
answers
70
views
Multi Node GPU Utilization with Docker for Machine Learning Platform
My home lab server has two nodes, let's call them Node A and Node B. Node A has 4 GPUs and Node B has 2 GPUs. How can I combine that computational power with Docker containers for machine learning/...
0
votes
1
answer
77
views
Cannot Successfully Implement Parallel Reduction for muti-CUDA GPU
I try to run the following code which would compute the dot product of two vectors, and the code can run well when the input number of GPU is 1, that is, the Omp package isn't really used, but when ...
0
votes
0
answers
217
views
Pytorch distribute process across nodes and gpu
I have access to 2 nodes, each has 2 GPUs. I want to have 4 processes, each has a GPU. I use nccl (if this is relevant).
Here is the Slurm script I tried. I tried different combinations of setup.
It ...
0
votes
0
answers
113
views
Same seed across different gpus in multiple workers in huggingface/pytorch
This is more of a discussion question because I have a workaround to suit my use case.
My understanding was if I am training my model on multiple GPUs (say n GPUs) in a DistributedDataParallel fashion,...
0
votes
0
answers
221
views
FSDP with size_based_auto_wrap_policy freezes training
I am doing multi-node and multi-GPU training. Right now, I am optimizing my multi-GPU code for faster training.
torch.cuda.set_device(self.rank)
model = FSDP(model, use_orig_params=True, device_id=...
0
votes
0
answers
85
views
How to run NVSHMEM with slurm
I'm getting started with using NVSHMEM and I wanted to start from a simple example, with not much success.
#include <nvshmem.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
//...
0
votes
0
answers
80
views
Accessing multiple GPUs on different hosts using LSF
I am using a HPC cluster with LSF Resource manager. The task is to train a tensorflow model. The graphics queue gq (Max jobs 96) has two hosts host1 with 4 GPUs and host2 with 3 GPUs. I want to use ...
0
votes
0
answers
293
views
CUDA out of memory while using pytorch lightning on multi-gpus
I try to finetune a diffusion model on RTX3090(24GB)x4,the details of the model are below.I use pytorch-lightning version 2.1 .I set the trainer with precision=16, accelerator="auto", ...
1
vote
1
answer
204
views
Getting NAN in loss function when training with multi gpu setup in tensorflow
I am recently training a deep learning model with multi GPU setup(dual 3090ti) with tf.distribute.MirroredStrategy().
In loss function, I have two types of loss; One is waveform loss and another is ...
1
vote
1
answer
159
views
Weird PyTorch Multiprocessing Error Where Main Loop Is Not Defined In __main__ | Kaggle
The following PyTorch code for single-node multi-GPU training with DDP seen here:
https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu.py
Gives the following error ...
0
votes
0
answers
156
views
Problem with torch.nn.DataParallel - data is distributed, but not the model, it seems
Expected to train model on 4 GPUs but 3/4 of the data disappears(?) somewhere
Looked all over other issues but to no avail - so asking my question here.
I am trying to train the model on 4 GPUs using ...
0
votes
0
answers
65
views
Uneven Multiple GPUs usage using Tensorflow
I am training a model using tensorflow, somehow only the first two GPU were used heavily, occupying 10000 MiB and 9290 MiB, respectively.
I have eight GPUs, the six other GPUs were used but only 300 ...
0
votes
0
answers
37
views
How to interpret multi-gpu tensorflow profile run to figure out bottleneck?
I am trying to figure out why my multi-gpu training using tensorflow MirroredStrategy is not scaling for training a 20 block x 128 filters ResNet. The single GPU run is scaling 100% with no gaps and ...
0
votes
0
answers
38
views
Issues with DataLoader Reinstantiation and Resource Cleanup in Optuna Trials
Description
I am using Optuna for hyperparameter optimization in my project. However, I've encountered a problem where DataLoader instances seem to accumulate over multiple trials. Despite creating a ...
1
vote
0
answers
36
views
Very strange timing in Nvidia Visual profiler
I'm trying to measure some programs, including some of those included in the CUDA samples, with nvprof in my environment with two Nvidia Quadro Q1000 GPUs.
For 5_Domain_Specific/MonteCarloMultiGPU ...
2
votes
0
answers
1k
views
Why does my device_map="auto" in transformers.pipline uses CPU only even though GPUs are available?
I came across this problem when trying out LLaMa 2 (13B version) on a 8X32GB-GPU server. The pipeline setting is like below:
pipeline = transformers.pipeline(
"text-generation",
...
0
votes
1
answer
65
views
Moving tensors between GPUs in Lightning
How can I move tensors from one gpu to another in training_step of pl.LightningModule?
In torch-based pipeline I use the following function to move tensors during multi-gpu fitting procedure:
def ...
1
vote
2
answers
364
views
cuDNN Error: CUDNN_STATUS_BAD_PARAM. Can someone tell what does this mean, why this is occurring?
I am trying to train an object detection model with yolov4 using multiple GPUs (tesla T4), but between around 1000 or 2000 iterations it's giving the below error.
(next mAP calculation at 1214 ...
0
votes
0
answers
29
views
Google Compute Engine Tesla K80 correct image
I am using Google compute Engine to do some ML. I want 8 GPUs and the cheapest is the Tesla K80.
I have tried many combinations of image and found every time that the GPUs are Unclaimed and so ...
2
votes
0
answers
749
views
What is the proper way to do multi-GPU inference with Pytorch?
I'm facing some issues with multi-GPU inference using pytorch and pytorch-lightning models.
At inference time, I need to use two different models in an auto-regressive manner. Since the auto-...
0
votes
0
answers
179
views
TypeError: cannot pickle '_thread.lock' object when wrapping Pytorch Distributed Data Parallel in a class
Here is a minimal code:
import threading
import torch
class DistributedMission:
def __init__(self):
self.lock = threading.Lock()
self.world_size = 8
def start(self):
...
0
votes
0
answers
70
views
Gathering result using Distributed Data Parallel needs too much video memory
the following is my task method used for multi-GPU predicting.
def worker(self, rank:int, pickle_file_name:str):
torch.distributed.init_process_group('nccl', rank=rank, world_size=self.world_size)
...
0
votes
0
answers
192
views
My Pytorch multi-GPU prediction code freezes at `spawn()`
The following small code does multi-GPU prediction using Pytorch.
I launch multiple tasks using torch.multiprocessing.spawn().
Inside task, I put no real prediction code. Just putting something into a ...
0
votes
0
answers
241
views
How can I train a yolov8 model with multi-gpu and rect=True?
I need to train a YOLOv8 model to detect objects in some extremely flat and elongated images. But I find that 'rect' mode is incompatible with multiple GPUs, it warns me that 'rect' mode is ...
1
vote
0
answers
125
views
Keras multi-GPU training, GPU resource can't be shared, causing training failure
I tried to apply keras multi-GPU to train my model with the following code:
tf.keras.backend.clear_session()
strategy = tf.distribute.MirroredStrategy()
print("Number of devices: {}".format(...
0
votes
0
answers
106
views
Communication using PyTorch distributed package extremely slow
I am using pytorch distributed package to write a simple program that let multiple GPUs on a single server communicate with each other. Here is the code:
def run(rank, world_size):
# torch.cuda....
0
votes
0
answers
8
views
Interference between Pyro4 and DDP process communication
I'm running a distributed machine learning training loop inside the remote function. The distributed training code is using pytorch-lightning and communication between the calling function and remote ...
0
votes
2
answers
1k
views
YoloV7 - Multi-GPU constantly gives RunTime Error
I am using YoloV7 to run a training session for custom object detection. My environment is as follows:
OS: Ubuntu 22.04
Python : 3.10
Torch Version : '2.1.0+cu121'
I am using AWS EC2 - g5.2xlarge ...
1
vote
0
answers
275
views
How to train a model on a cluster with multiple GPUs (single node)
I wish to train a model on a multiple-GPUs, 1-Node cluster on AML (Azure ML).
I have my cluster (STANDARD_ND40RS_V2, which has 8 V100)
I have my Dataset, my Environment (docker), and my script.
When ...
0
votes
0
answers
66
views
Problem with running a tensorflow-based script on multiple GPUs of a cluster
I want to train a 3D model based on tensorflow. I'm using a node in a cluster with 8 GPUs (Geforce GTX 1080 ti, with 1045 MB memory for each one). My inputs are 3D CT images. When I reduce the image ...
0
votes
1
answer
524
views
How to clear GPU memory after training each fold
I am getting an error after a few folds of training is done..
The error states:
Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.15GiB (rounded to 17338417152)requested by op _EagerConst
...
1
vote
0
answers
147
views
torch DDP Multi-GPU gives low accuracy metric
I am trying Multi-GPU, single machine DDP training in PyTorch (CIFAR-10 + ResNet-18 setup). You can refer to the model architecture code here and the full training code here.
Within main() function, ...