6

This is a more conceptual question, but I have to confess I have been dealing with it for a while.

Suppose you want to train a neural network (NN), using for instance keras. As it is recommended you perform previous to the training a normalization or standardization of the data, so, for instance, with a standardization:

x_new = (x_old - mean)/standarddev

Then, you carry on the training (model.fit in keras) and minimize the loss function, all very nice.

Edit: In my case, I have a set of values between 200 and 400. It's a NN with 1 input, 1 output. I standardize as told, the input values AND the expected values, so the NN learns the weights and biases in a standardized way.

Now, imagine that I have a completely new dataset of values between 200 and 400 and I want to predict an output, using the NN with the previous training. You can use model.predict(x) in keras, with x the completely new set of values I have received, standardized (or normalized) because your NN was trained in that way. But then, what I get, after the predict is an array of values standardized, but I want to map them to the usual range of 200 to 400. And I don't know how to do this.

I know you can carry on the training without normalizing or standardizing, but I have read that if you standardize (or normalize), with values in the range of the output of the units (neurons) (for instance, between 0 and 1 for a sigmoid), the training improves.

Thank you.

2
  • Ok - let's check if I understood : You got your data, which you put to neural network, in a normalized form - and you want to recover its original values? Or you want to destandarize your output? And - btw. what is the output of your function? Some code might be useful. Jun 3, 2016 at 8:45
  • Yes, I have an x array of inputs and an y array of outputs, with values between 200 and 400. They are both normalized and the training performed. After that I have a new array, x_2, with values between 200 and 400, and I want to predict an outcome using predict in keras. The problem is that the NN have been trained in a normalized way. If I normalize this new array x_2 what I get, let's call it, y_2 is normalized, and what I want is denormalize, to get values between 200 and 400.
    – David
    Jun 3, 2016 at 9:03

3 Answers 3

Reset to default

Trending sort

Trending sort is based off of the default sorting method — by highest score — but it boosts votes that have happened recently, helping to surface more up-to-date answers.

It falls back to sorting by highest score if no posts are trending.

6

Ok, I think that I got what is your problem correctly so I will try to explain you how to deal with data normalization :

1. Assumption about distribiution of inputs and outputs : usually in neural network training - what you assume is that your data (both input and output) comes from some probability distribiutions : let's call it X for input and Y of output. There are some reasons to make this distribiution to be zero mean and with unit standard deviation during the training phase.

2. Statistical part of data normalization and recovery : because of that - you have to solve another task during training your network. This task is to estimate the mean and standard deviation of both input distribution X and output distribution Y. You are doing that by simply applying empirical mean and standard deviation to your training data.

3. Application phase - inputs : when you apply your model to new input you are also assuming that your input comes from distribiution X so you also need to standarize it to be zero mean and unit standard deviation and here is a funny part - you can use both training set and a set of new data to obtain even better estimation of mean and standard deviation of X but to avoid overfitting in validation case - you usually use the mean and standard deviation obtained during training phase to make new data standarized.

4. Application phase - outputs : this part is trickier because when you apply your network to new standarized inputs you get new outputs from Y* ~ (Y - mean'(Y)) / sd'(Y) where mean'(Y) and sd'(Y) are estimation of mean and standard deviation obtained empirically from your training set and Y is original distribiution of your output. It's because during your training set you feed your optimizer with output data from this distribiution. So to make your outputs to be restandarized you need to apply transformation: Y* * sd'(Y) + mean'(Y). which is reverse to standarization transformation.

SUMMARY:

Your training and application phase looks following :

  1. You are obtaining statistics needed for both training phase and application phase by computing empirical mean and standard deviation of your training inputs (mean'(X) and sd'(X) and empirical mean and standard deviation of your outputs (mean'(Y) and sd'(Y)). It's important to store them because they will be needed in application phase.
  2. You standarize your both input and output data to be zero mean and unit standard deviation and train your model on them.
  3. During application phase you standarize your inputs by subtracting it by stored mean'(X) and dividing by stored sd'(X) to obtain new output Y*
  4. You destandarize your outputs using stored mean'(Y) and sd'(Y) - obtained during training phase - by transformation (Y* * sd'(Y) + mean'(Y).

I hope that this answer will solve your problem and leave you with no doubts about details of standarization and destandarization of your data :)

3
  • Ok, Marcin, I think this solves my problem, thank you. I know how to apply it, but I think I still have a conceptual doubt: why do I have to always use the mean'(X) and mean'(Y) (and the same for std) of the training phase? That is, it seems strange to me becasue you can use the mean of the new dataset to predict, (let's say mean''(X_new)), which is closer to the actual mean of the input dataset of prediction.
    – David
    Jun 6, 2016 at 8:52
  • 1
    There are at least three reasons : 1. Both new and old values should come from the same distribution - so it doesn't matter. 2. Your model was learnt on an old data. 3. You know the performance of your model with the old normalization. Taking statistics from a new data might be risky due to bias in your new data set which you are not able to recognize. Jun 6, 2016 at 11:01
  • Ok, Marcin. I think I understand. Thank you for everything.
    – David
    Jun 6, 2016 at 11:05
5

You standarized your input/output values with the following formula:

X_s = (X - mean) / std

To destandarize, you must have the mean and std values for input and outputs. Save them somewhere and then use the following equation:

X = X_s * std + mean

For example, let's say that for your [200, 400] range, the mean is 300 and the standard deviation is 100. Then say, for a normalized value of 0.5, the unnormalized value is:

X = 0.5 * 100 + 300 = 350

If you didn't store the mean/std then you have no way of recovering the original values.

2
  • Yes, I know that. You can do that to destandarize after the training phase. The problem is that with a new dataset to make predictions you have to use a new mean and a new standard deviation, or perhaps the old ones? That was my doubt. But thank you for your answer.
    – David
    Jun 6, 2016 at 8:41
  • 2
    @David you use the ones that were used when you trained the network, this is, what you call old ones.
    – Dr. Snoopy
    Jun 6, 2016 at 10:10
1

Depending on if you standarize your outputs or not :

1. If not : then your output values are not standarized and you don't have to worry about it.

2. If yes: then you keep your mean/sd and unstadarize your output simply by (output * sd) + mean.

1
  • Marcin, thank you for your answer but I am afraid I don't fully understand you. I am going to edit my question to make it clearer.
    – David
    Jun 3, 2016 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.