53

tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None) outputs random values from a normal distribution.

tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None) outputs random values from a truncated normal distribution.

I tried googling 'truncated normal distribution'. But didn't understand much.

3 Answers 3

80

The documentation says it all: For truncated normal distribution:

The values are drawn from a normal distribution with specified mean and standard deviation, discarding and re-drawing any samples that are more than two standard deviations from the mean.

Most probably it is easy to understand the difference by plotting the graph for yourself (%magic is because I use jupyter notebook):

import tensorflow as tf
import matplotlib.pyplot as plt

%matplotlib inline  

n = 500000
A = tf.truncated_normal((n,))
B = tf.random_normal((n,))
with tf.Session() as sess:
    a, b = sess.run([A, B])

And now

plt.hist(a, 100, (-4.2, 4.2));
plt.hist(b, 100, (-4.2, 4.2));

enter image description here


The point for using truncated normal is to overcome saturation of tome functions like sigmoid (where if the value is too big/small, the neuron stops learning).

4
  • thanks for the code. It is great. however, i can't print out histogram on the pc as shown below. Do you know why? ,>>> plt.hist(b,100,(-4.2,4.2)); (array([ 3.00000000e+00, 6.00000000e+00, 9.00000000e+00, 4.00000000e+00, 1.60000000e+01, 8.00000000e+00, 3.864, 3.948, 4.032, 4.116, 4.2 ]), <a list of 100 Patch objects>)
    – Hong
    Jun 20, 2017 at 15:48
  • @Hong sorry, but I have no idea why. It looks like it was calculated but for some reason is not plotted. May be you can ask a question related to matplotlib Jun 20, 2017 at 16:54
  • @SalvadorDali A question following this one would be then, when is one more preferred than the other? It seems to me that truncated_normal is more often than not the preferred option. When is random_normal a valid choice? Jun 26, 2017 at 14:57
  • 7
    @CarlosJimenezBermudez it is hard to predict which initialization will lead to the fastest learning. random_normal might be used with RELU activations. The reason for not-using random_normal is saturations for sigmoid/tanh, but they are not frequently used now. Jun 26, 2017 at 19:31
25

tf.truncated_normal() selects random numbers from a normal distribution whose mean is close to 0 and values are close to 0. For example, from -0.1 to 0.1. It's called truncated because your cutting off the tails from a normal distribution.

tf.random_normal() selects random numbers from a normal distribution whose mean is close to 0, but values can be a bit further apart. For example, from -2 to 2.

In machine learning, in practice, you usually want your weights to be close to 0.

3
  • 1
    In ML, the usual suggestion is that weights have a mean of 0, standard deviation of 0.1 or 0.01, be close to 0 and have a uniform distribution. Can you tell exactly why ?
    – pravsels
    May 24, 2017 at 12:34
  • @pravbeatle I believe this post can help you, cs231n.github.io/neural-networks-2. Those values are preferred because the network will train faster.
    – Kenan
    May 24, 2017 at 18:58
  • @pravbeatle another explanation of weight saturation for sigmoid link: stats.stackexchange.com/questions/228670/…
    – weiheng
    Dec 8, 2017 at 12:10
8

The API documentation for tf.truncated_normal() describes the function as:

Outputs random values from a truncated normal distribution.

The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.