1

I can change the value of a tensor that requires grad without autograd knowing about it:

def error_unexpected_way_to_by_pass_safety():
    import torch 
    a = torch.tensor([1,2,3.], requires_grad=True)
    # are detached tensor's leafs? yes they are
    a_detached = a.detach()
    #a.fill_(2) # illegal, warns you that a tensor which requires grads is used in an inplace op (so it won't be recorded in computation graph so it wont take the right derivative of the forward path as this op won't be in it)
    a_detached.fill_(2) # weird that this one is allowed, seems to allow me to bypass the error check from the previous comment...?!
    print(f'a = {a}')
    print(f'a_detached = {a_detached}')
    a.sum().backward()

this throws no errors. Though, I am able to change the contents of a which is a tensor that requires grad without autograd knowing about. Which means the computation graph does not know about this op (filling with 2). This seems wrong. Can anyone shed light what is going on?

2 Answers 2

5

.detach gives you a view on the same data, so modifying the data of the detached tensor modifies the data of the original. You can check this like so:

a.data_ptr() == a_detached.data_ptr() # True

As for why this is how .detach is implemented (as opposed to doing a defensive copy), that's a design question that only the PyTorch authors know the answer to. I assume that it's to save unnecessary copies, but users then need to be aware that they have to copy the tensors themselves if they want to modify the detached ones in-place.

Note that you can also alter the non-detached tensor if you really want to:

a.data.fill_(2)

PyTorch isn't trying to stop you from "hacking" autograd; users still have to be aware of how to use tensors properly so that gradients will be tracked correctly.

3
  • I thought .data field was going to be removed according to what I read in the Pytorch forum. Is that not correct? can you really do a.data.fill_(2) still? Jun 17, 2020 at 20:19
  • does id(a.data) give you the memory address of the tensor data? Is that different from id(a)? Jun 17, 2020 at 20:19
  • I'm using PyTorch version 1.5.0, and I can certainly still do a.data.fill_, but I don't know about future plans for this; you may be more informed there than I am. Yes to both the questions from your second comment (also, as you'd expect now, id(a.data) == id(a_detached.data) even though id(a) != id(a_detached))
    – Nathan
    Jun 18, 2020 at 1:50
1

Adding to the existing answer here. The reason detach doesn't copy the data is definitely to save unnecessary copies - if you want to have a full copy you can always have a.clone().detach() version of a (or a.detach().clone()). You can do just one of these (e.g. just clone or just detach) and all of these sense in some situations.

The most important reason one would want to use detach without clone is because this is the way to implement the so-called "StopGradient" operation in pytorch (stop_gradient in tf). Imagine the situation when you want to use tensor a in your NN twice in such a way that gradients propagate through in one case and don't propagate in another (and no one is expected to modify the tensor in-place).

As to clone'ing without detach - it seems a bit unusual, but I've seen such examples like that (mostly people wanted to ensure original tensor won't be updated, but gradients will propagate to it).

Modifying tensors in-place is usually something you want to avoid (except optimizer steps). In any case you'll have to exercise extreme caution while doing so since it is an easy way to render computation graph invalid w.r.t. gradient computation.

7
  • 1
    you mean detach? not clone? Clone does copy the data afaik... Jun 17, 2020 at 20:21
  • key comment! (and no one is expected to modify the tensor in-place). Jun 17, 2020 at 20:22
  • Yes this is the main reason I've seen clone. Referencing your comment: "As to clone'ing without detach - it seems a bit unusual, but I've seen such examples like that (mostly people wanted to ensure original tensor won't be updated, but gradients will propagate to it)." Jun 17, 2020 at 20:22
  • I learned recently that in-place ops are tracked by autograd. Does that mean that it also tracks the ops by the optimizer step? If yes, then what is the use of the higher library at all? Jun 17, 2020 at 20:24
  • 1
    I'm not a big expert on how exactly autograd tracking in-place ops. Most cases I've seen were when autograd noticed that tensor was changed in place after it was used for some computation and halted the whole process to avoid error computation. As of higher - it is actually explicitly avoiding in-place optimizer step by creating new version of parameters every time user wants to make a step with "differentiable optimizer" (e.g. in higher.innerloop_ctx). Jun 17, 2020 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.