In PyTorch, the Tensor
class has a grad_fn
attribute. This references the operation used to obtain the tensor: for instance, if a = b + 2
, a.grad_fn
will be AddBackward0
. But what does "reference" mean exactly?
Inspecting AddBackward0
using inspect.getmro(type(a.grad_fn))
will state that the only base class of AddBackward0
is object
. Additionally, the source code for this class (and in fact, any other class which might be encountered in grad_fn
) is nowhere to be found in the source code!
All of this leads me to the following questions:
- What precisely is stored in
grad_fn
and how is it called during back-propagation? - How come the objects that get stored in
grad_fn
do not have some sort of common super class, and why is there no source code for them on GitHub?