10

I have for some time gotten pretty bad results using the tool keras, and haven't been suspisous about the tool that much.. But I am beginning to be a bit concerned now.

I tried to see whether it could handle a simple XOR problem, and after 30000 epochs it still haven't solved it...

code:

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import SGD
import numpy as np

np.random.seed(100)

model = Sequential()
model.add(Dense(2, input_dim=2))
model.add(Activation('tanh'))
model.add(Dense(1, input_dim=2))
model.add(Activation('sigmoid'))
X = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
y = np.array([[0],[1],[1],[0]], "float32")
model.compile(loss='binary_crossentropy', optimizer='adam')
model.fit(X, y, nb_epoch=30000, batch_size=1,verbose=1)

print(model.predict_classes(X))

Here is part of my result:

4/4 [==============================] - 0s - loss: 0.3481     
Epoch 29998/30000
4/4 [==============================] - 0s - loss: 0.3481     
Epoch 29999/30000
4/4 [==============================] - 0s - loss: 0.3481     
Epoch 30000/30000
4/4 [==============================] - 0s - loss: 0.3481     
4/4 [==============================] - 0s
[[0]
 [1]
 [0]
 [0]]

Is there something wrong with the tool - or am I doing something wrong??

Version I am using:

MacBook-Pro:~ usr$ python -c "import keras; print keras.__version__"
Using TensorFlow backend.
2.0.3
MacBook-Pro:~ usr$ python -c "import tensorflow as tf; print tf.__version__"
1.0.1
MacBook-Pro:~ usr$ python -c "import numpy as np; print np.__version__"
1.12.0

Updated version:

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import Adam, SGD
import numpy as np

#np.random.seed(100)

model = Sequential()
model.add(Dense(units = 2, input_dim=2, activation = 'relu'))
model.add(Dense(units = 1, activation = 'sigmoid'))
X = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
y = np.array([[0],[1],[1],[0]], "float32")
model.compile(loss='binary_crossentropy', optimizer='adam')
print model.summary()
model.fit(X, y, nb_epoch=5000, batch_size=4,verbose=1)

print(model.predict_classes(X))
7
  • try increasing the epoch to 50000
    – Aditya
    Commented May 3, 2017 at 3:01
  • Possible duplicate of How to use keras for XOR
    – Aditya
    Commented May 3, 2017 at 3:02
  • @ADITYA It converges around here.. I tried with mse. Its a bit weird to use mse for a classification, but yes it helped, but still 0.1255 loss.
    – J.Down
    Commented May 3, 2017 at 3:05
  • 1
    Lower loss does not necessarily means that things got better. Also, the thread pointed out by @ADITYA uses 4 hidden units. They got it wrong, the network didn't learn XOR (as pointed out by J.Down in the other answer). Commented May 3, 2017 at 4:35
  • what is your learning rate? Commented May 3, 2017 at 8:16

4 Answers 4

3
+50

I cannot add a comment to Daniel's response as I don't have enough reputation, but I believe he's on the right track. While I have not personally tried running the XOR with Keras, here's an article that might be interesting - it analyzes the various regions of local minima for a 2-2-1 network, showing that higher numerical precision would lead to fewer instances of getting stuck on a gradient descent algorithm.

The Local Minima of the Error Surface of the 2-2-1 XOR Network (Ida G. Sprinkhuizen-Kuyper and Egbert J.W. Boers)

On a side note I won't consider using a 2-4-1 network as over-fitting the problem. Having 4 linear cuts on the 0-1 plane (cutting into a 2x2 grid) instead of 2 cuts (cutting the corners off diagonally) just separates the data in a different way, but since we only have 4 data points and no noise in the data, the neural network that uses 4 linear cuts isn't describing "noise" instead of the XOR relationship.

1

I think it's a "local minimum" in the loss function.

Why?

I have run this same code over and over for a few times, and sometimes it goes right, sometimes it gets stuck into a wrong result. Notice that this code "recreates" the model every time I run it. (If I insist on training a model that found the wrong results, it will simply be kept there forever).

from keras.models import Sequential
from keras.layers import *
import numpy as np

m = Sequential()
m.add(Dense(2,input_dim=2, activation='tanh'))
#m.add(Activation('tanh'))

m.add(Dense(1,activation='sigmoid'))
#m.add(Activation('sigmoid'))

X = np.array([[0,0],[0,1],[1,0],[1,1]],'float32')
Y = np.array([[0],[1],[1],[0]],'float32')

m.compile(optimizer='adam',loss='binary_crossentropy')
m.fit(X,Y,batch_size=1,epochs=20000,verbose=0)
print(m.predict(X))

Running this code, I have found some different outputs:

  • Wrong: [[ 0.00392423], [ 0.99576807], [ 0.50008368], [ 0.50008368]]
  • Right: [[ 0.08072935], [ 0.95266515], [ 0.95266813], [ 0.09427474]]

What conclusion can we take from it?

The optimizer is not dealing properly with this local minimum. If it gets lucky (a proper weight initialization), it will fall in a good minimum, and bring the right results.

If it gets unlucky (a bad weight initialization), it will fall in a local minimum, without really knowing that there are better places in the loss function, and its learn_rate is simply not big enough to escape this minimum. The small gradient keeps turning around the same point.

If you take the time to study which gradients appear in the wrong case, you will probably see it keeps pointing towards that same point, and increasing the learning rate a little may make it escape the hole.

Intuition makes me think that such very small models have more prominent local minimums.

10
  • changing tanh to sigmoid or relu would be sensible, but even running 10 times, I still get error prediction.
    – J.Down
    Commented May 16, 2017 at 2:52
  • Have you assured you "recreated" the model each time? Commented May 16, 2017 at 2:54
  • I noticed you have set a random seed = 100. Doesn't that affect keras as well? Wouldn't that make your weight initalization be always the same? I suppose you'd find different results using different seeds (or simply no seed at all, but that would probably require you to restart the kernel). Commented May 16, 2017 at 4:18
  • Not according to this: github.com/fchollet/keras/issues/439 So all the weights should be different, for everytime i run it.
    – J.Down
    Commented May 16, 2017 at 5:07
  • 1
    Actually I was able to learn the pattern by your network. I trained it for 1000 epochs 25 times and I was able to achieve 100% of accuracy. This suggests that you are simply stacking in a poor minimum. Try it on your own by simply running a fit for 1000 epochs in a loop 20-30 times. Commented May 16, 2017 at 14:15
1

Instead of just increasing the number of epochs, try using relu for the activation of your hidden layer instead of tanh. Making only that change to the code you provide, I am able to obtain the following result after only 2000 epochs (Theano backend):

import numpy as np
print(np.__version__) #1.11.3
import keras
print(theano.__version__) # 0.9.0
import theano
print(keras.__version__) # 2.0.2

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import Adam, SGD

np.random.seed(100)

model = Sequential()
model.add(Dense(units = 2, input_dim=2, activation = 'relu'))
model.add(Dense(units = 1, activation = 'sigmoid'))
X = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
y = np.array([[0],[1],[1],[0]], "float32")
model.compile(loss='binary_crossentropy', optimizer='adam'
model.fit(X, y, epochs=2000, batch_size=1,verbose=0)
print(model.evaluate(X,y))
print(model.predict_classes(X))
4/4 [==============================] - 0s
0.118175707757
4/4 [==============================] - 0s
[[0]
[1]
[1]
[0]]

It would be easy to conclude that this is due to vanishing gradient problem. However, the simplicity of this network suggest that this isn't the case. Indeed, if I change the optimizer from 'adam' to SGD(lr=0.01, momentum=0.0, decay=0.0, nesterov=False) (the default values), I can see the following result after 5000 epochs with tanh activation in the hidden layer.

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import Adam, SGD

np.random.seed(100)

model = Sequential()
model.add(Dense(units = 2, input_dim=2, activation = 'tanh'))
model.add(Dense(units = 1, activation = 'sigmoid'))
X = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")
y = np.array([[0],[1],[1],[0]], "float32")
model.compile(loss='binary_crossentropy', optimizer=SGD())
model.fit(X, y, epochs=5000, batch_size=1,verbose=0)

print(model.evaluate(X,y))
print(model.predict_classes(X))
4/4 [==============================] - 0s
0.0314897596836
4/4 [==============================] - 0s
[[0]
 [1]
 [1]
 [0]]

Edit: 5/17/17 - Included complete code to enable reproduction

4
  • You are right.. The usage of 'tanh' would indeed make it harder to set the weights accurately. I tried with 'relu' and SGD - which you according to this post have tried, but is still get inaccurate results. Even after 5000 epochs.
    – J.Down
    Commented May 7, 2017 at 23:22
  • Which version of Keras are you using? I'm using Keras 2.0.2, Theano 0.9.0 and Numpy 1.11.3.
    – dhinckley
    Commented May 8, 2017 at 1:38
  • Added it in my post. But... Keras: 2.0.3 tensorflow: 1.0.1 numpy:1.12.0
    – J.Down
    Commented May 8, 2017 at 1:42
  • I've updated my answer to include the complete python code.
    – dhinckley
    Commented May 17, 2017 at 20:28
0

The minimal neuron network architecture required to learn XOR which should be a (2,2,1) network. In fact, if maths shows that the (2,2,1) network can solve the XOR problem, but maths doesn't show that the (2,2,1) network is easy to train. It could sometimes takes a lot of epochs (iterations) or does not converge to the global minimum. That said, I've got easily good results with (2,3,1) or (2,4,1) network architectures.

The problem seems to be related to the existence of many local minima. Look at this 1998 paper, «Learning XOR: exploring the space of a classic problem» by Richard Bland. Furthermore weights initialization with random number between 0.5 and 1.0 helps to converge.

It works fine with Keras or TensorFlow using loss function 'mean_squared_error', sigmoid activation and Adam optimizer. Even with pretty good hyperparameters, I observed that the learned XOR model is trapped in a local minimum about 15% of the time.

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from tensorflow.keras import initializers
import numpy as np 

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([[0],[1],[1],[0]])

def initialize_weights(shape, dtype=None):
    return np.random.normal(loc = 0.75, scale = 1e-2, size = shape)

model = Sequential()
model.add(Dense(2, 
                activation='sigmoid', 
                kernel_initializer=initialize_weights, 
                input_dim=2))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='mean_squared_error', 
              optimizer='adam', 
              metrics=['accuracy'])

print("*** Training... ***")

model.fit(X, y, batch_size=4, epochs=10000, verbose=0)

print("*** Training done! ***")

print("*** Model prediction on [[0,0],[0,1],[1,0],[1,1]] ***")

print(model.predict_proba(X))

*** Training... ***

*** Training done! ***

*** Model prediction on [[0,0],[0,1],[1,0],[1,1]] ***

[[0.08662204] [0.9235283 ] [0.92356336] [0.06672956]]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.