2 of 2
added 848 characters in body
vis7
  • 101
  • 8

keras cnn model only predicts one class for all the test images

I am trying to build an image classification model with 2 classes with (1) or without (0). I can build the model and get an accuracy of 1. which is too good to be true (which is an issue) but when I use predict_generator as I have my images in folders, it only returns 1 class 0 (without class). There seems to be an issue but I can't work it out, I have looked at a number of articles but I still can't fix the issue.

image_shape = (220, 525, 3) #height, width, channels
img_width = 96
img_height = 96
channels = 3

epochs = 10

no_train_images = 11957              #!ls ../data/train/* | wc -l
no_test_images = 652                 #!ls ../data/test/* | wc -l
no_valid_images = 6156               #!ls ../data/test/* | wc -l

train_dir = '../data/train/'
test_dir = '../data/test/'
valid_dir = '../data/valid/'


test folder structure is the following:
test/test_folder/images_from_both_classes.jpg


#!ls ../data/train/without/ | wc -l 5606        #theres no class inbalance
#!ls ../data/train/with/ | wc -l 6351

#!ls ../data/valid/without/ | wc -l 2899
#!ls ../data/valid/with/ | wc -l 3257

classification_model = Sequential()

# First layer with 2D convolution (32 filters, (3, 3) kernel size 3x3, input_shape=(img_width, img_height, channels))
classification_model.add(Conv2D(32, (3, 3), input_shape=input_shape))
# Activation Function = ReLu increases the non-linearity
classification_model.add(Activation('relu'))
# Max-Pooling layer with the size of the grid 2x2
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
# Randomly disconnets some nodes between this layer and the next 
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(32, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.25))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.3))

classification_model.add(Flatten())
classification_model.add(Dense(64))
classification_model.add(Activation('relu'))
classification_model.add(Dropout(0.5))
classification_model.add(Dense(1))
classification_model.add(Activation('sigmoid'))

# Using binary_crossentropy as we only have 2 classes
classification_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])



batch_size = 32

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    zoom_range=0.2)

# this is the augmentation configuration we will use for testing:
# only rescaling
valid_datagen = ImageDataGenerator(rescale=1. / 255)
test_datagen = ImageDataGenerator()

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = True)

valid_generator = valid_datagen.flow_from_directory(
    valid_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = False)

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size = (img_width, img_height),
    batch_size = 1,
    class_mode = None,
    shuffle = False)

mpd = classification_model.fit_generator(
    train_generator,
    steps_per_epoch = no_train_images // batch_size,         # number of images per epoch
    epochs = epochs,                                         # number of iterations over the entire data
    validation_data = valid_generator,
    validation_steps = no_valid_images // batch_size)  

Epoch 1/10 373/373 [==============================] - 119s 320ms/step - loss: 0.5214 - acc: 0.7357 - val_loss: 0.2720 - val_acc: 0.8758

Epoch 2/10 373/373 [==============================] - 120s 322ms/step - loss: 0.2485 - acc: 0.8935 - val_loss: 0.0568 - val_acc: 0.9829

Epoch 3/10 373/373 [==============================] - 130s 350ms/step - loss: 0.1427 - acc: 0.9435 - val_loss: 0.0410 - val_acc: 0.9796

Epoch 4/10 373/373 [==============================] - 127s 341ms/step - loss: 0.1053 - acc: 0.9623 - val_loss: 0.0197 - val_acc: 0.9971

Epoch 5/10 373/373 [==============================] - 126s 337ms/step - loss: 0.0817 - acc: 0.9682 - val_loss: 0.0136 - val_acc: 0.9948

Epoch 6/10 373/373 [==============================] - 123s 329ms/step - loss: 0.0665 - acc: 0.9754 - val_loss: 0.0116 - val_acc: 0.9985

Epoch 7/10 373/373 [==============================] - 140s 376ms/step - loss: 0.0518 - acc: 0.9817 - val_loss: 0.0035 - val_acc: 0.9997

Epoch 8/10 373/373 [==============================] - 144s 386ms/step - loss: 0.0539 - acc: 0.9832 - val_loss: 8.9459e-04 - val_acc: 1.0000

Epoch 9/10 373/373 [==============================] - 122s 327ms/step - loss: 0.0434 - acc: 0.9850 - val_loss: 0.0023 - val_acc: 0.9997

Epoch 10/10 373/373 [==============================] - 125s 336ms/step - loss: 0.0513 - acc: 0.9844 - val_loss: 0.0014 - val_acc: 1.0000

valid_generator.batch_size=1
score = classification_model.evaluate_generator(valid_generator, 
                                                no_test_images/batch_size, pickle_safe=False)
test_generator.reset()
scores=classification_model.predict_generator(test_generator, len(test_generator))
    
print("Loss: ", score[0], "Accuracy: ", score[1])

predicted_class_indices=np.argmax(scores,axis=1)
print(predicted_class_indices)

labels = (train_generator.class_indices)
labelss = dict((v,k) for k,v in labels.items())
predictions = [labelss[k] for k in predicted_class_indices]

filenames=test_generator.filenames
results=pd.DataFrame({"Filename":filenames,
                      "Predictions":predictions})

print(results)

Loss: 5.404246180551993e-06 Accuracy: 1.0

#print(predicted_class_indices) - ALL 0 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

                              Filename Predictions
0      test_folder/video_3_frame10.jpg   without
1    test_folder/video_3_frame1001.jpg   without
2    test_folder/video_3_frame1006.jpg   without
3    test_folder/video_3_frame1008.jpg   without
4    test_folder/video_3_frame1009.jpg   without
5    test_folder/video_3_frame1010.jpg   without
6    test_folder/video_3_frame1013.jpg   without
7    test_folder/video_3_frame1014.jpg   without
8    test_folder/video_3_frame1022.jpg   without
9    test_folder/video_3_frame1023.jpg   without
10    test_folder/video_3_frame103.jpg   without
11   test_folder/video_3_frame1036.jpg   without
12   test_folder/video_3_frame1039.jpg   without
13    test_folder/video_3_frame104.jpg   without
14   test_folder/video_3_frame1042.jpg   without
15   test_folder/video_3_frame1043.jpg   without
16   test_folder/video_3_frame1048.jpg   without
17    test_folder/video_3_frame105.jpg   without
18   test_folder/video_3_frame1051.jpg   without
19   test_folder/video_3_frame1052.jpg   without
20   test_folder/video_3_frame1054.jpg   without
21   test_folder/video_3_frame1055.jpg   without
22   test_folder/video_3_frame1057.jpg   without
23   test_folder/video_3_frame1059.jpg   without
24   test_folder/video_3_frame1060.jpg   without

...just some of the outputs but all 650+ are without class.

This is the output and as you can see all the predicted values are 0 for the without class.

This is my first attempt at using Keras and CNN so any help would be really appreciated.

UPDATE

I have solved this. I am currently working on the accuracy but the main problem is now solved.

This is the line that caused problems.

predicted_class_indices=np.argmax(scores,axis=1)

argmax would be returning the index position of the result but as I was using binary classes and in my final layer I had 1 dense. It will only return a single value so it will always return the first class (0 as the index position). As the network is only set, to return one class.

Changing the following fixed my issue.

  1. Changed the class_mode to 'categorical' for the train and test generators
  2. Changed the final dense layer from 1 to 2 so this will return scores/probabilities for both classes. So when you use argmax, it will return the index position of the top score indicating which class it has predicted.
vis7
  • 101
  • 8