1

I'm taking a course on PyTorch. And I'm wondering why we need to separately tell to torch.utils.data.DataLoader output on what device it's running on. If the model is already on CUDA why doesn't it automatically change the inputs accordingly? This pattern seems funny to me:

model.to(device)

for ii, (inputs, labels) in enumerate(trainloader):

    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

Is there a use case where I'd like to have model running on GPU but my inputs to be on CPU mode or vice versa?

1 Answer 1

7

When you call model.to(device) (assuming device is a GPU) your model parameters will be moved to your GPU. Regarding to your comment: they are moved from CPU memory to GPU memory then.

By default newly created tensors are created on CPU, if not specified otherwise. So this applies also for your inputs and labels.

The problem here is that all operands of an operation need to be on the same device! If you leave out the to and use CPU tensors as input you will get an error message.

Here is an short example for illustration:

import torch

# device will be 'cuda' if a GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# creating a CPU tensor
cpu_tensor = torch.rand(10)
# moving same tensor to GPU
gpu_tensor = cpu_tensor.to(device)

print(cpu_tensor, cpu_tensor.dtype, type(cpu_tensor), cpu_tensor.type())
print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())

print(cpu_tensor*gpu_tensor)

Output:

tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619]) torch.float32 <class 'torch.Tensor'> torch.FloatTensor
tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619], device='cuda:0') torch.float32 <class 'torch.Tensor'> torch.cuda.FloatTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ac794171c178> in <module>()
     12 print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())
     13 
---> 14 print(cpu_tensor*gpu_tensor)

RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'other'
3
  • What does it mean they are on GPU - maybe you have an article link to share? =D Also we have torch.no_grad why not have torch.to('CUDA') and all things will be put on GPU?
    – Mihkel L.
    Dec 9, 2018 at 18:10
  • 1
    @MihkelL. Meant is the ram (random access memory). So GPU has it's on RAM. So there is CPU RAM, if they "are on CPU" they are stored in CPU Memory, if they "are on GPU" they are stored in GPU RAM. I hope this makes sense to you! (Terms RAM and Memory are used as synonyms here)
    – MBT
    Dec 9, 2018 at 18:12
  • You can also take a look here: quora.com/…
    – MBT
    Dec 9, 2018 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.