I looked to the following examples from Keras:
MLP in MNIST: https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py
CNN in MNIST: https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py
I run both in Theano on CPU. In the MLP I have a mean time of approximately 16s per epoch with a total of 669,706 parameters:
Layer (type) Output Shape Param #
=================================================================
dense_33 (Dense) (None, 512) 401920
_________________________________________________________________
dropout_16 (Dropout) (None, 512) 0
_________________________________________________________________
dense_34 (Dense) (None, 512) 262656
_________________________________________________________________
dropout_17 (Dropout) (None, 512) 0
_________________________________________________________________
dense_35 (Dense) (None, 10) 5130
=================================================================
Total params: 669,706.0
Trainable params: 669,706.0
Non-trainable params: 0.0
In the CNN, I eliminated the last hidden layer from the original code. I also changed the optimizer to rmsprop to make both cases comparable, leaving the following architecture:
Layer (type) Output Shape Param #
=================================================================
conv2d_36 (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
conv2d_37 (Conv2D) (None, 24, 24, 64) 18496
_________________________________________________________________
max_pooling2d_17 (MaxPooling (None, 12, 12, 64) 0
_________________________________________________________________
dropout_22 (Dropout) (None, 12, 12, 64) 0
_________________________________________________________________
flatten_17 (Flatten) (None, 9216) 0
_________________________________________________________________
dense_40 (Dense) (None, 10) 92170
=================================================================
Total params: 110,986.0
Trainable params: 110,986.0
Non-trainable params: 0.0
However, the average time here is of approximately 340 s per epoch! Even though there are six times less parameters!
To check more on this, I reduced the number of filters per layer to 4, leaving the following architecture:
Layer (type) Output Shape Param #
=================================================================
conv2d_38 (Conv2D) (None, 26, 26, 4) 40
_________________________________________________________________
conv2d_39 (Conv2D) (None, 24, 24, 4) 148
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 12, 12, 4) 0
_________________________________________________________________
dropout_23 (Dropout) (None, 12, 12, 4) 0
_________________________________________________________________
flatten_18 (Flatten) (None, 576) 0
_________________________________________________________________
dense_41 (Dense) (None, 10) 5770
=================================================================
Total params: 5,958.0
Trainable params: 5,958.0
Non-trainable params: 0.0
Now the time is of 28 s per epoch even though there are roughly 6000 parameters!!
Why is this? Intuitively, the optimization should only depend on the number of variables and the calculation of the gradient (which due to same batch size should be similar).
Some light on this? Thank you