When running a PyTorch training program with num_workers=32
for DataLoader
, htop
shows 33 python process each with 32 GB of VIRT
and 15 GB of RES
.
Does this mean that the PyTorch training is using 33 processes X 15 GB = 495 GB of memory? htop
shows only about 50 GB of RAM and 20 GB of swap is being used on the entire machine with 128 GB of RAM. So, how do we explain the discrepancy?
Is there a more accurate way of calculating the total amount of RAM being used by the main PyTorch program and all its child DataLoader worker processes?
Thank you
VIRT
inhtop
roughly refers to the amount of RAM your process has access to. WhereasRES
is the actual RAM consumed. From my understanding,RES
is something that's based on the parent process – so look at theRES
usage of the parent (set yourself to tree view) to get a rough idea of how much RAM you're using, total.nvidia-smi
would also be a good proxy in terms of GPU memory.