11

Final Edit: Cleaned up the question and accepted runDOSrun's answer. IVlad's is equally good, and user3760780's is extremely helpful too. I recommend reading all three of those as well as the comments. The TLDR answer is that Possibility #1 is more or less the correct one but I phrased it very badly.

What does the input layer consist of in Neural Networks? What does that layer do?

A similar question is here Neural Networks: Does the input layer consist of neurons? but the answers there did not clear up my confusion.

Like the poster in the question above, I'm confused by the many contradicting things the Internet has to say about the input layer of a basic feed-forward network.

I'll skip the links to contradicting tutorials and articles and list the three possibilities that I can see. Which one (if any) is the correct one?

  1. The input layer passes the data directly to the first hidden layer where the data is multiplied by the first hidden layer's weights.
  2. The input layer passes the data through the activation function before passing it on. The data is then multiplied by the first hidden layer's weights.
  3. The input layer has its own weights that multiply the incoming data. The input layer then passes the data through the activation function before passing it on. The data is then multiplied by the first hidden layer's weights.

Thanks!

EDIT 1: Here is an image and an example for further clarity.

4
  • what's your question? Sep 11, 2015 at 3:21
  • 1
    I just want to know what is your specifically confusion... Sep 11, 2015 at 3:23
  • @AlvaroJoao I'm confused about what the input layer does/what it consists of. The three listed options in the question cover all the possibilities (I think) so instead of asking a complicated question with diagrams and such, I decided to just list the 3 options and ask which one was correct. Thanks for your help! (I added another couple sentences at the beginning of the question to clarify)
    – MindSeeker
    Sep 11, 2015 at 3:26
  • I really hope that I had helped you. @MindSeeker Sep 11, 2015 at 3:54

4 Answers 4

7

Out of your 3 descriptions, the first one fits best:

  1. The input layer passes the data directly to the first hidden layer where the data is multiplied by the first hidden layer's weights.

A standard Multilayer Perceptron's input layer consists of units (you can call them input neurons, but I prefer to use the term units because you expect a neuron to do some computations, which isn't the case for the input layer) that you assign a value to (a part of one of your input data instances, or the value of a feature of a single instance in machine learning terms) and they simply feed that value to every neuron in the first hidden layer, resulting in exactly the first case you portray in your image.

I would rephrase it to this for more accuracy:

  • Each unit of the input layer, in top-to-bottom order, passes its assigned value to each neuron of the first hidden layer. Then, each hidden layer neuron multiplies each of these values (x1, x2, ..., xm) with its weight vector (w1, w2, ..., wm), sums the multiplied values (x1*w1 + x2*w2 + ... + xm*wm), applies its activation function to this sum (logistic, tanh, identity function) and returns the value computed by the activation function to the next layer.

So for your example, top-most neuron in the hidden layer would receive the inputs:

.5, .6

From the input layer, and it would compute and return:

g(.4 * .5 + .3 * .6)

Where g is its activation function, which can be anything:

g(x) = x # identity function, like in your picture
g(x) = 1 / (1 + exp(-x)) # logistic sigmoid

In my opinion, it is not entirely right to say the weights also go into it, since its weights are its own, but I guess this distinction is not very important; it certainly doesn't affect the result.

You have to remember that this is all conceptually speaking. In proper implementations, you won't have any actual layers at all, just some matrix multiplications. But they will implement the same concept. When trying to understand something, you should start by referring to the underlying concept.

  1. The input layer passes the data through the activation function before passing it on. The data is then multiplied by the first hidden layer's weights.

This is not correct, the input layer only returns some values assigned to it to every neuron in the next layer.

Is there some reference where you found it? I'm quite sure it's not standard practice to do this.

  1. The input layer has its own weights that multiply the incoming data. The input layer then passes the data through the activation function before passing it on. The data is then multiplied by the first hidden layer's weights.

Again, not the case. It has no weights and no activation functions.

6
  • Thanks for the answer! If I'm understanding you correctly, Possibility 1 is close but is still not correct. I've added a Possibility 4 to illustrate what I think you mean. It seems unlikely to me (I explained why above) but it's very possible I misunderstood your answer. If I'm totally wrong, an illustration in terms of the example above (inputs .5 and .6) would help unambiguously illustrate exactly what you mean. Thanks for your time and continued patience!
    – MindSeeker
    Sep 11, 2015 at 15:48
  • 1
    @MindSeeker no, possiblity 1 is it. You portray it correctly in the picture. I would just rephrase the text for a bit more clarity, I've posted how. Let me know if you have any more questions.
    – IVlad
    Sep 11, 2015 at 16:01
  • @MindSeeker added a bit more.
    – IVlad
    Sep 11, 2015 at 16:06
  • 1
    Just a small addendum/correction: There's actually many object-oriented implementations used by the research community that have layers (sometimes even units) as actual objects to bind the code closer to the concepts which is often helpful to experiment with more complex network models.
    – runDOSrun
    Sep 11, 2015 at 16:26
  • @runDOSrun I didn't know that. More complex models would need very fast implementations, because they'd need to be tested on large data sets in order to prove that they're better. Even with fast GPU implementations, people sometimes train for days, so I think I'd prefer just doing it efficiently from the beginning. I can see the pedagogical value in an OOP implementation, but not much research value.
    – IVlad
    Sep 11, 2015 at 16:39
5

Since I gave an answer in the thread you linked I'll try my best to clear up your confusions as well.

First thing I notice is that you seem to be confused about which layer a weight belongs to. The answer is not to one but two. The weight in your image is the weight from input to hidden layer and should be referenced as such in order to avoid ambiguity within multiple layers. Again, different conventions. But stick to this one since it reflects the official math notations best (weights are referenced as w_ij indicating that a weight goes from i to j (sometimes j to i depending on the author)).

Let me also start by saying that natural speech and graphs are always ambiguous and the best way to approach things is math. It's plain and clear ...although most of us may have a bad relation with it :)

That being said let's start with an image anyway (this is a single layer perceptron, just pretend the next layer is actually a hidden layer it makes no difference):

enter image description here

This image is clearer for beginners since it breaks up the process of activating a single neuron into all its components:

  • The inputs and weights (between inp and hid layer) are combined and summed. This is the linear combination enter image description here with net_j being the input for the neuron j in the next hidden layer.

  • This net input is fed into the activation function f, such that the activation per hidden neuron in the hidden layer is enter image description here (here described as o_j, I'll refer to it as h_j since we pretend it's in a hidden layer).

So the whole process of getting the value for each hidden neuron h_j can be summed up with the simple formula:

enter image description here

  • Each input gets multiplied with its connection to this neuron h_j.
  • All these weight-input products get summed.
  • This is fed into the activation function f.
  • The result is the value of the neuron h_j.

This is done with all neurons h_j and then repeated for the next layer.

So actually none of your options are 100% correct or complete. 1.) is phrased correctly but incomplete.

Edit: The correct possibility in your image is #1:

enter image description here

(Weights only have 2 indices as said, units have 1 index. w_ij is the weight from unit x_i to h_j)

2
  • Thanks for the detailed and complete answer! So, if I'm understanding correctly, in terms of my example above, the correct answer is the same is as Possibility #1 where (.4*.5 + .3*.6) goes into the top first-hidden-layer neuron's activation function, correct? In other words, Possibility #1 got the results right in the image I provided, but used bad terminology/phrasing in it's definition?
    – MindSeeker
    Sep 11, 2015 at 16:22
  • 1
    Yep, you got it. h_1 = f(x_1 * w_11 + x_2 * w_21) = f(.4*.5 + .3*.6)
    – runDOSrun
    Sep 11, 2015 at 16:30
1

The standard approach would be to first apply a linear transformation to your input data, i.e. to "apply the weights" (that might also be a convolution). By doing that you get a new matrix of values. Then you apply an activation function (non-linearity) to that. Your first possibility seems to match that. (Your 3rd apparently too, as the input layer seems to act as a combination of linear transformation and non-linearity, which is identical to having a separate layer for that.)

Applying a non-linearity directly to the input is likely not a good idea as the network did not get the chance to project the input into a better space. E.g. if you choose a ReLU activation function (max(0, value)) as your first transformation then any input value below 0 will be lost, which would not be the case if you had a linear transformation before that.

Applying two linear transformations in a row (input -> apply weights -> apply weights) is not a good idea either, because those could be merged to one linear transformation (and the network should be able to learn that), i.e. two linear transformations in a row is a waste of computation.

2
  • Thanks for your help! I think I understand you fully, but I want to be sure. Are you saying "Possibility 1 is the correct answer, watch out for these other alternatives, but Possibility 1 is 100% correct" or are you saying "Possibility 1 is almost right, but you need to do X differently"? If I've completely misunderstood you, a solution in terms of the example above (inputs .5 and .6) would help disambiguate. Thanks again for your help and time!
    – MindSeeker
    Sep 11, 2015 at 15:58
  • 1
    Possibility 1 would be the standard way of implementing a neural net. Everything else in my post is just intended to describe why it is the standard choice.
    – aleju
    Sep 11, 2015 at 16:18
0

Basically these 3 options say the same thing.

I will try to explain in another works:

The first layer just tells you how your data looks like or what is relevant for the neural network.

For example:

You have collected every day for one year the millimeters of rain in your city.

The data will looks like this:

PS: 0 represents no rain.

mm 0 0 0 0.1 0.2 0 . . .

Now comes the next year. And you want to predict the next days' millimeters.

You will use a neural network for it.

your input layer will be this data which has only one attribute: millimeters. (only one node)

Coming back to your options, they may vary in small details like:

3 The input layer has its own weights that multiply the incoming data.

that seems confusing, cos the other options didn't said anything about this own weights.

But you have to remember that there's a LOT of implementations and approaches for neural networks. But you have to always focus on the basics concepts:

  1. The input layer, gets the input data and pass throw the hidden layers
  2. The output will bring you the processed data, maybe a prediction as the example or a classification.
  3. Everything that happens between that will depends on the approach or the technique you want to use

PS(2): There are few implementations of Neural Networks(NN) that attends to a specific type of databases or problem. Do not try to generalize everything.

4
  • @MindSeeker tell me if helped you Sep 11, 2015 at 4:17
  • Thanks for your answer! I posted an example to clarify. It demonstrates that the 3 options do very different things. Also, I understand that there are many different "correct" implementations, but I am just starting, and am trying to get the basics down before I start working on my own implementations. The tutorials that I was trying to follow (both are quite vague on this point) are ai-junkie.com/ann/evolved/nnt6.html and takinginitiative.wordpress.com/2008/04/03/…. Thanks again for your help, I look forward to your updated answer!
    – MindSeeker
    Sep 11, 2015 at 5:02
  • Actually, there aren't that many types of neural networks, especially the kind the OP is referring to: Multilayer Perceptrons. And there are certainly no differences between how the input layer is handled within this class of ANNs. The literature is very clear about the definitions and the terminology. The input layer does not map anything.
    – IVlad
    Sep 11, 2015 at 12:26
  • @IVlad about the definitions are really vague. You may fit the recurrent MLP to meet the definitions, for example. Sep 11, 2015 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.