I want to write some custom CUDA kernels for neural networks to speed up computations, but I don't want to spend time differentiating tensor expressions by hand if there are packages that can do it automatically.
Is there a python package that can show expression for symbolic matrix differentiation?
I know sympy
can do it for non-matrix expressions like this:
def func(x):
return 1 / x
arg_symbols = sp.symbols(inspect.getfullargspec(func).args)
sym_func = func(*arg_symbols)
s = ''
for arg in arg_symbols:
s += '{}\n'.format(arg, sp.Lambda(arg_symbols, sym_func.diff(arg)))
# this is what I need:
print(s)
>>> Lambda(x, -1/x**2)
I know autograd
package can compute the derivatives of matrix expressions
After the function is evaluated, autograd has a list of all operations that were performed and which nodes they depended on. This is the computational graph of the function evaluation. To compute the derivative, we simply apply the rules of differentiation to each node in the graph.
But is there a way to get this differentiation computational graph from it or some similar package?