Questions tagged [activation-function]
Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.
0
votes
1answer
27 views
Using a custom step activation function in Keras results in “An operation has `None` for gradient.” error. How to resolve this?
I am building auto-encoder and I want to encode my values into a logical matrix. However, when I'm using my custom step activation function in one of the intermediate layers (all other layers are ...
-2
votes
1answer
33 views
Which Activation Function to use for Neural Networks
Apologies in advance if this question is not the conventional approach, where a snippet of code or a question about a code is involved. I'm just trying to understand certain specific points on the ...
0
votes
1answer
17 views
keras custom activation to drop under certain conditions
I am trying to drop the values less than 1 and greater than -1 in my custom activation like below.
def ScoreActivationFromSigmoid(x, target_min=1, target_max=9) :
condition = K.tf.logical_and(K....
0
votes
1answer
24 views
keras custom activation function with threshold replacement
For the custom activation function that changes to scores below, I want to replace the values with the activated_x with less than threshold=0.5 to be 0.
How can I modify?
def ...
1
vote
1answer
69 views
How to make a custom activation function in tensorflow
I need to make an activation function which is not exist in tensorflow.How should I do? I ever saw this link,
How to make a custom activation function with only Python in Tensorflow?
but I still don't ...
0
votes
2answers
42 views
Piecewise activation function in tensorflow and broadcasting math operation
I am trying to implement and test an activation function that I have read in a paper.
I am using Keras with tensorflow backend and I want to feed the activation function to the fit method of my model....
1
vote
3answers
47 views
How does ReLu work with zero-centered output domain?
In the problem i am trying to solve, my output domain is zero centered, between -1 an 1. When looking up activation functions i noticed that ReLu outputs values between 0 and 1, which basically would ...
1
vote
2answers
32 views
Role of activation function in calculating the cost function for artificial neural networks
I have some difficulty with understanding the role of activation functions and cost functions. Lets take a look at a simple example. Lets say I am building a neural network (artificial neural network)....
1
vote
2answers
38 views
why linear function is useless in multiple layer neural network? How last layer become the linear function of the input of first layer?
I was studying about activation function in NN but could not understand this part properly -
"Each layer is activated by a linear function. That activation in turn goes into the next level as input ...
0
votes
1answer
37 views
Neural Network pruning mechanism
I am working on SqueezeNet pruning . I have some questions regarding the pruning code which is based on the paper : PRUNING CONVOLUTIONAL NEURAL NETWORKS FOR RESOURCE EFFICIENT INFERENCE
def ...
0
votes
1answer
21 views
Getting the output prior to non-linear activation in Keras
How can I get the value prior to activation when I use the following syntax to define a layer in Keras:
model.add(Convolution2D(128, 5, 5, activation='relu'))
I know that I can simply use:
model....
0
votes
0answers
29 views
How to use Softmax Activation Function in Emgucv ANN_MLP?
How to use Softmax Activation Function in Emgucv ANN_MLP?
Is there a way to use custom Activation function in Emgucv ANN_MLP?
0
votes
0answers
37 views
Neural network solving “RGB” trivial problem - can it even solve such problem?
Background
I am coding basic feed forward neural network with backpropagation. I am at state that it solves some of trivial problems such as XOR or x>y, so everything looks fine. I wanted to code ...
0
votes
1answer
221 views
How to implement RBF activation function in Keras?
I am creating a customized activation function, RBF activation function in particular:
from keras import backend as K
from keras.layers import Lambda
l2_norm = lambda a,b: K.sqrt(K.sum(K.pow((a-b),...
0
votes
1answer
28 views
Derivative of activation function vs partial derivative wrt. loss function
Some terms in AI are confusing me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function?
These terms are confusing: derivative ...
1
vote
1answer
15 views
Block Activation Function Realization in Tensorflow
I was trying to reproduce a DNN where a block activation function called BlockRelu is used. It is defined as
BlockRelu
I tried to write this function according to some example codes about self-...
0
votes
0answers
6 views
How can I extract information regarding the activations in each layer from a keras model?
I have a keras model in a '.h5' file and I want to extract details like the type of activation used in each layer and any attributes those activation might have. Is there a way to do it. If there is a ...
0
votes
1answer
30 views
What is the difference between keras.activations.softmax and keras.layers.Softmax?
What is the difference between keras.activations.softmax and keras.layers.Softmax? Why are there two definitions of the same activation function?
keras.activations.softmax: https://keras.io/...
1
vote
1answer
182 views
Matrix multiplication in pyTorch
I'm writing a simple neural network in pyTorch, where features and weights both are (1, 5) tensors. What are the differences between the two methods that I mention below?
y = activation(torch.sum(...
0
votes
1answer
40 views
How to use maxout in Tensorflow?
guys! I have a question to ask.If I want to use maxout as activation function , how should I write the codes in Tensorflow? An input parameter is required in the slim.maxout() function, so it cannot ...
0
votes
1answer
51 views
using tanh as activation function in MNIST dataset in tensorflow
I am working on simple MLP neural network for MNIST dataset using tensorflow as my homework. in the question we should implement a multilayer perceptron with tanh as activation function. I should use ...
0
votes
0answers
13 views
my question is about multi class classification with keras for wifi data
**I want to do a multiclass classification using keras, I have a dataset with 1500 samples from 150 different labels( quite huge), data are signal, **
the result is very bad, starting from .02, ...
0
votes
1answer
62 views
How to plot keras activation functions in a notebook
I wanted to plot all Keras activation functions but some of them are not working. i.e. linear throws an error:
AttributeError: 'Series' object has no attribute 'eval'
which is weird. How can I ...
3
votes
1answer
280 views
Keras - Nan in summary histogram LSTM
I've written an LSTM model using Keras, and using LeakyReLU advance activation:
# ADAM Optimizer with learning rate decay
opt = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-...
1
vote
1answer
99 views
Custom activation with parameter
I'm trying to create an activation function in Keras that can take in a parameter beta like so:
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects
from keras....
0
votes
0answers
65 views
Normalization for MLPregressor
I'm using scikit-learn, and trying to understand how normalize my input variables, for different activation functions using MLPregressor:
Relu
Tanh
Logistic
How can i properly normalize the data for ...
1
vote
1answer
95 views
Can I use a Sigmoid activation for my output layer, even if my CNN model is doing a regression?
Final objective: Object Midpoint calculation.
I have a small dataset (around 120 images), which has an object (the same in all cases), and the labels are the normalized x,y coordinates of the ...
0
votes
0answers
28 views
implementing cost function + activation function -incorrect value returned
I'm working my way through a machine learning tutorial and I'm stuck on a section of logistic regression that requires calculating values of the activation and provide them to the cost function.
I'm ...
1
vote
1answer
99 views
Tensorflow, square root activation function implementation (shaping error)
For the purpose of implementing a classification NN I found some really useful tutorials, like this one (2 hidden layer, one-hot-encoding output, dropout regularization, normalization etc.) which ...
-2
votes
1answer
377 views
Why is ReLU a non-linear activation function?
As I understand it, in a deep neural network, we use an activation function (g) after applying the weights (w) and bias(b) (z := w * X + b | a := g(z)). So there is a composition function of (g o z) ...
0
votes
0answers
93 views
Inception Model mxnet on Raspberry Pi: fix an unknown activation type error?
I'm trying to implement the Inception model on a Raspberry Pi using mxnet, and getting an error that I haven't been able to unravel: "unknown activation type".
I successfully installed opencv, mxnet, ...
2
votes
0answers
48 views
XOR NN not learnable with 2 hidden nodes and sigmoid activation?
I felt like my backpropagation intuition wasn't crystal clear, so I wrote a neural network class to train/predict on XOR. It has 2 inputs, 1 output, a variable number of hidden nodes, and bias nodes ...
0
votes
0answers
7 views
Where does this picture come from? Step activation function vs. sigmoid activation function
does somebody knows where this picture comes from? It´s about step activation function and sigmoid activation function.activation functions
Thanks in advance!
0
votes
1answer
259 views
how to define the derivative of a custom activation function in keras
I have a custom activation function and its derivative, although I can use the custom activation function I don't know how to tell keras what is its derivative.
It seems like it finds one itself but ...
0
votes
2answers
79 views
what activation function should I use to enforce rounding like behaviour
I need an activation function that rounds my tensors.
the derivative(gradients) of the function round() is 0 (or None in tensorflow) which makes it unusable as an activation function.
I am looking ...
0
votes
2answers
33 views
Understanding of threshold value in a neural network
Consider the hypothetical neural network here
$o_1$ is the output of neuron 1.
$o_2$ is the output of neuron 2.
$w_1$ is the weight of connection between 1 and 3.
$w_2$ is the weight of ...
0
votes
2answers
102 views
Implementation of tanh() activation function for a CNN
I'm trying to implement the activation function tanh on my CNN, but it doesn't work, the result is always "NaN". So i created a simple application where i have a randomized matrix and try to apply the ...
-3
votes
1answer
43 views
Activation function for linear dataset
I have been working with data sets that mostly show the linear relationship between different attributes/features. What activation should I be using with linear datasets? I have been using sigmoid ...
1
vote
2answers
68 views
How does Keras optimize weights on layers which have no activation?
Background:
If I am not mistaken, when training a network we feed forward performing sigmoid(sum(W*x)) for every layer then in back-propagation we calculate the error and the deltas (change) then we ...
1
vote
1answer
849 views
Why is ReLU used in regression with Neural Networks?
I am following the official TensorFlow with Keras tutorial and I got stuck here: Predict house prices: regression - Create the model
Why is an activation function used for a task where a continuous ...
0
votes
0answers
9 views
Activation Function for a conv layer
Why does everyone use Relu for Conv layers? I understand that it makes training quicker; however, I was wondering if anyone has come across a paper that tried multiple activation functions for their ...
0
votes
0answers
160 views
Neural Chess: Sample neural network gets stuck at value
I am attempting to write a neural network to play chess, but I am running into problems with the output. I am using python-chess library, and built rewards in. The network has 4 outputs and three ...
1
vote
1answer
128 views
Why pytorch has two kinds of Non-linear activations?
Why pytorch has two kinds of Non-linear activations?
Non-liner activations (weighted sum, nonlinearity):
https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity
Non-...
0
votes
0answers
57 views
Which activation function should I use for my output layer if my output is a Glove vector representing a word
My output is 332 dimension (300 glove + 32 my custom vector)
values of this vector range from -1 to +1
I got horrible results using sigmoid as it confined the output to 0 to 1.
I'm trying Tanh right ...
0
votes
2answers
509 views
Is there a logit function in tensorflow?
Is there a logit function in tensorflow, i.e. the inverse of sigmoid function? I have searched google but have not found any.
-1
votes
1answer
79 views
How to set a suitable activation function for an ANN having negative input values
I am creating an ANN which has 3 input neurons which take inputs from the device "s accelerometer in the form of x , y , z. These values are positive as well as negative depending upon the ...
1
vote
2answers
805 views
How do I implement leaky relu using Numpy functions
I am trying to implement leaky Relu, the problem is I have to do 4 for loops for a 4 dimensional array of input.
Is there a way that I can do leaky relu only using Numpy functions?
2
votes
0answers
108 views
Differing results for MNIST autoencoder due to different placement of activation function
I stumbled across a strange phenomenon while playing around with variational autoencoders. The problem is quite simple to describe:
When defining the loss function for the VAE, you have to use some ...
-1
votes
1answer
30 views
When does ReLU kills the neurons?
I am confused regarding the dying ReLU problem. ReLU will kill the neuron only during the forward pass? Or also during the backward pass?
1
vote
1answer
1k views
Keras - Default Axis for softmax function is set to Axis
I am learning how to create sequential models. I have a model:
*model = Sequential()*
I then went on to add pooling layers and convolution layers (which were fine). But when creating the dense ...