I'm using a neural network made of 4 input neurons, 1 hidden layer made of 20 neurons and a 7 neuron output layer.
I'm trying to train it for a bcd to 7 segment algorithm. My data is normalized 0 is -1 and 1 is 1.
When the output error evaluation happens, the neuron saturates wrong. If the desired output is 1
and the real output is -1
, the error is 1-(-1)= 2
.
When I multiply it by the derivative of the activation function error*(1-output)*(1+output)
, the error becomes almost 0
Because of 2*(1-(-1)*(1-1)
.
How can I avoid this saturation error?