1

I have to do a CNN to detect Diabetic retinopathy in 4th stage (it have to detect if there DR in 4th stage or not, doesn't require detect the other levels). The input will be images like this : https://i.stack.imgur.com/OSfgn.jpg

To be better to classificate, i'm refining my image: https://i.stack.imgur.com/s6bsu.png

So, I have a database with 700 images of retinas in level 0, and 700 retinas with level 4.

The problem is all model I tried to make isn't work, generally it became a overfitting problem..

I already tried to use Sequential model, Functional API.. and in one question I've made here, a user recommended me to use VGG16 >> question : https://datascience.stackexchange.com/questions/60706/how-do-i-handle-with-my-keras-cnn-overfitting

And now, I'm trying to use VGG16 but still doens't work, all my predictions are 0 and I have no idea what to do to handle it..

This is my train.py:

import cv2
import os
import numpy as np

from keras.layers.core import Flatten, Dense, Dropout, Reshape
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import Conv2D
from keras.layers.pooling import MaxPooling2D
from keras import regularizers
from keras.models import Model
from keras.layers import Input, ZeroPadding2D, Dropout
from keras import optimizers
from keras.optimizers import SGD
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.utils import to_categorical 

from keras.applications.vgg16 import VGG16

# example of using a pre-trained model as a classifier
from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.applications.vgg16 import preprocess_input
from keras.applications.vgg16 import decode_predictions

TRAIN_DIR = 'train/'
TEST_DIR = 'test/'
v = 'v/'
BATCH_SIZE = 32
NUM_EPOCHS = 5

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    # Get all subdirectories
    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]
    
    # Loop over each directory
    for File in FolderList:
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            # Convert the path into a file
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
            #ImageCV[index]= np.array(ImageCV[index]) / 255.0
            LabelList.append(classes.index(os.path.splitext(File)[0])) 
            
            ImageCV[index] = cv2.addWeighted(ImageCV[index],4, cv2.GaussianBlur(ImageCV[index],(0,0), 224/30), -4, 128)

    return ImageCV, LabelList

data, labels = ReadImages(TRAIN_DIR)
valid, vlabels = ReadImages(TEST_DIR)

vgg16_model = VGG16(weights="imagenet", include_top=True)
 
# (1) visualize layers
print("VGG16 model layers")
for i, layer in enumerate(vgg16_model.layers):
    print(i, layer.name, layer.output_shape)

# (2) remove the top layer
base_model = Model(input=vgg16_model.input, 
                   output=vgg16_model.get_layer("block5_pool").output)

# (3) attach a new top layer
base_out = base_model.output
base_out = Reshape((25088,))(base_out)
top_fc1 = Dropout(0.5)(base_out)
# output layer: (None, 5)
top_preds = Dense(1, activation="sigmoid")(top_fc1)

# (4) freeze weights until the last but one convolution layer (block4_pool)
for layer in base_model.layers[0:14]:
    layer.trainable = False

# (5) create new hybrid model
model = Model(input=base_model.input, output=top_preds)

# (6) compile and train the model
sgd = SGD(lr=1e-4, momentum=0.9)
model.compile(optimizer=sgd, loss="binary_crossentropy", metrics=["accuracy"])

datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(data)

# fits the model on batches with real-time data augmentation:
model.fit_generator(datagen.flow(np.array(data), np.array(labels), batch_size=32), 
                    steps_per_epoch=len(np.array(data)) / 32, epochs=5)


#history = model.fit([data], [labels], nb_epoch=NUM_EPOCHS, 
#                    batch_size=BATCH_SIZE, validation_split=0.1)

# evaluate final model
#vlabels = model.predict(np.array(valid))

model.save('model.h5')

When I run it, returns a accuracy of ~1.0 or 0.99 % with a minimal loss ~0.01..

This is my predict.py:

from keras.models import load_model
import cv2
import os
import json
import h5py
import numpy as np
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input

TEST_DIR = 'v/'

def fix_layer0(filename, batch_input_shape, dtype):
    with h5py.File(filename, 'r+') as f:
        model_config = json.loads(f.attrs['model_config'].decode('utf-8'))
        layer0 = model_config['config']['layers'][0]['config']
        layer0['batch_input_shape'] = batch_input_shape
        layer0['dtype'] = dtype
        f.attrs['model_config'] = json.dumps(model_config).encode('utf-8')

fix_layer0('model.h5', [None, 224, 224, 3], 'float32')

model = load_model('model.h5')

for filename in os.listdir(r'v/'):
    if filename.endswith(".jpg") or filename.endswith(".ppm") or filename.endswith(".jpeg") or filename.endswith(".png"):
        ImageCV = cv2.resize(cv2.imread(os.path.join(TEST_DIR) + filename), (224,224))
        
        x = image.img_to_array(ImageCV)
        x = np.expand_dims(x, axis=0)
        x = preprocess_input(x)
        print(np.argmax(model.predict(x)))

When I run it, all my predictions are 0.. and if a drop the 'np.argmax' and run only model.predict, returns the follow result:

[[0.03993018]]
[[0.9984968]]
[[1.]]
[[1.]]
[[0.]]
[[0.9999999]]
[[0.8691623]]
[[1.01611796e-07]]
[[1.]]
[[0.]]
[[1.]]
[[0.17786741]]

Considering that the 2 first images are class 0 and the others are class 1 (level 4), the results aren't 0.99 or 1.0 of acc..

What should I have to do? I really, strongly appreciate any help!

UPDATE

I've updated my code as @Manoj has said.. I've added validation and early stopping:

es = EarlyStopping(monitor='val_loss', verbose=1)

# fits the model on batches with real-time data augmentation:
model.fit_generator(datagen.flow(np.array(data), np.array(labels), batch_size=32), 
                    steps_per_epoch=len(np.array(data)) / 32, epochs=5,
                    validation_data=(np.array(valid), np.array(vlabels)),
                    nb_val_samples=72, callbacks=[es])

And returns these numbers:

Epoch 1/5
44/43 [==============================] - 452s 10s/step - loss: 0.2377 - acc: 0.9162 - val_loss: 1.9521 - val_acc: 0.8472
Epoch 2/5
44/43 [==============================] - 445s 10s/step - loss: 0.0229 - acc: 0.9991 - val_loss: 1.8908 - val_acc: 0.8611
Epoch 3/5
44/43 [==============================] - 447s 10s/step - loss: 0.0107 - acc: 0.9993 - val_loss: 1.7658 - val_acc: 0.8611
Epoch 4/5
44/43 [==============================] - 458s 10s/step - loss: 0.0090 - acc: 0.9993 - val_loss: 1.6805 - val_acc: 0.8750
Epoch 5/5
44/43 [==============================] - 463s 11s/step - loss: 0.0052 - acc: 0.9993 - val_loss: 1.6730 - val_acc: 0.8750

But after that my predictions (that were 7/12 correct) now is 5/12 correct..

What I can do to handle it?

UPDATE 2

I've putted this code in my train.py:

mean = datagen.mean  
std = datagen.std

print(mean, "mean")
print(std, "std")

and the values returned by these prints I inserted in predict.py:

def normalize(x, mean, std):
    x[..., 0] -= mean[0]
    x[..., 1] -= mean[1]
    x[..., 2] -= mean[2]
    x[..., 0] /= std[0]
    x[..., 1] /= std[1]
    x[..., 2] /= std[2]
    return x

for filename in os.listdir(r'v/'):
    if filename.endswith(".jpg") or filename.endswith(".ppm") or filename.endswith(".jpeg") or filename.endswith(".png"):
        ImageCV = cv2.resize(cv2.imread(os.path.join(TEST_DIR) + filename), (224,224))
        
        x = image.img_to_array(ImageCV)
        x = np.expand_dims(x, axis=0)
        x = normalize(x, [59.5105,61.141457,61.141457], [60.26705,61.85445,63.139835])

        prob = model.predict(x)
        if prob < 0.5:
            print("nonPDR")
        else:
            print("PDR")
        print(filename)

and now all my predictions are (class 1) PDR... I've made something wrong?

UPDATE 3

I've drop out the gaussianblur I was using in ReadImages, and included the follow:

data = np.asarray(data)
valid = np.asarray(valid)

data = data.astype('float32')
valid = valid.astype('float32')

data /= 255
valid /= 255

And after run my train.py:

Epoch 1/15

44/43 [==============================] - 476s 11s/step - loss: 0.7153 - acc: 0.5788 - val_loss: 0.6937 - val_acc: 0.5556

Epoch 2/15

44/43 [==============================] - 468s 11s/step - loss: 0.5526 - acc: 0.7275 - val_loss: 0.6838 - val_acc: 0.5833

Epoch 3/15

44/43 [==============================] - 474s 11s/step - loss: 0.5068 - acc: 0.7595 - val_loss: 0.6927 - val_acc: 0.5694

Epoch 00003: early stopping

After, I update the std and mean on predict.py:

for filename in os.listdir(r'v/'):
    if filename.endswith(".jpg") or filename.endswith(".ppm") or filename.endswith(".jpeg") or filename.endswith(".png"):
        ImageCV = cv2.resize(cv2.imread(os.path.join(TEST_DIR) + filename), (224,224))
        
        ImageCV = np.asarray(ImageCV)
        
        ImageCV = ImageCV.astype('float32')
        
        ImageCV /= 255  
        x = ImageCV
        
        x = np.expand_dims(x, axis=0)
        x = normalize(x, [0.12810835, 0.17897758, 0.23883381], [0.14304605, 0.18229756, 0.2362126])
        
        prob = model.predict(x)
        if prob <= 0.70: # I CHANGE THE THRESHOLD TO 0.7
            print("nonPDR >>>", filename)
            nonPdr += 1
        else:
            print("PDR >>>", filename)
            pdr += 1
        print(prob)
print("Number of retinas with PDR: ",pdr)
print("Number of retinas without PDR: ",nonPdr)

And after run this code, I'm getting roughly 75% accuracy in my test dir..

So, can I improve something, or this is the maximum for these tiny number of images?

4
  • Changing the optimizer, increase the learning rate and momentum with a strong decay factor may help your overfitting problem. That's all I can say with limited time but I will have a closer look today to help you out.
    – Physicing
    Sep 30, 2019 at 13:23
  • Furthermore you can add EarlyStopping with more epoches trained. Seems like you're doing it for 5 epoches.
    – Physicing
    Sep 30, 2019 at 14:44
  • Provide links to the jpg/png images, not the gallery on imgur Sep 30, 2019 at 15:05
  • I've updated the post
    – user12096782
    Sep 30, 2019 at 19:14

1 Answer 1

0
  1. The preprocessing steps done on the data should be same for training and test. I see at least two inconsistencies. First, on the train data, GaussianBlur is applied to all the images. Usually, such transformations are used as data augmentation strategies and not applied to the entire training set. Second, the normalization used for training and test should be same. In the code snippets above, for predictions the vgg16.preprocess_input is applied which uses the mean/variance of imagenet data while during training the mean/variance is calculated from the training data itself. What you can do is take the datagen.mean and datagen.std values after calling datagen.fit and use it during predictions for normalizing the data instead of preprocess_input.

  2. You don't have a validation generator defined. When training, you use a training set & validation set, and stop the training when the validation loss doesn't improve. Otherwise, the model will overfit to the training dataset.

    https://gist.github.com/fchollet/7eb39b44eb9e16e59632d25fb3119975 https://keras.io/callbacks/#earlystopping

  3. Since the final layer of your network is a sigmoid like this

    top_preds = Dense(1, activation="sigmoid")(top_fc1)

    there is only one output and it's a probability value from 0 to 1. np.argmax is not relevant here.

    np.argmax is used when the last layer uses softmax activation with two outputs whose probabilities sum to 1 and the index with the higher probability is chosen as the result.

    Coming back to the results you obtain with sigmoid, usually a threshold is chosen to decide whether to classify it as class 0 or class 1. The default threshold is 0.5. A ROC curve can be created using the probabilities to come up with the optimal threshold.

    https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html

    Using threshold of 0.5,

    prob = model.predict(x)
    if prob < 0.5:
        output = 0
    else:
        output = 1
    
    [[0.03993018]] => < 0.5, class 0 correct
    [[0.9984968]]  => > 0.5, class 1 incorrect
    [[1.]]         => > 0.5, class 1 correct
    [[1.]]         => > 0.5, class 1 correct
    [[0.]]         => < 0.5, class 0 incorrect
    [[0.9999999]]  => > 0.5, class 1 correct
    [[0.8691623]]  => > 0.5, class 1 correct
    [[1.01611796e-07]] => < 0.5, class 0 incorrect
    [[1.]]             => > 0.5, class 1 correct
    [[0.]]             => < 0.5, class 0 incorrect
    [[1.]]             => > 0.5, class 1 correct
    [[0.17786741]]     => < 0.5, class 0 incorrect
    

    Accuracy = 7/12 = 58%

8
  • I've generated other model with validation and earlystop as you've said, but the predictions became worse ... It predicted correct only in 5 samples
    – user12096782
    Sep 30, 2019 at 19:04
  • Hi... You only answer 1 time?
    – user12096782
    Oct 1, 2019 at 20:24
  • std = datagen.std(data), mean = datagen.mean(data) something like this?
    – user12096782
    Oct 2, 2019 at 11:46
  • datagen.mean and datagen.std are numpy arrays containing the mean and std. Oct 2, 2019 at 11:49
  • and how I call it in my predict.py?
    – user12096782
    Oct 2, 2019 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.