24

How to compute the mean IU (mean Intersection over Union) score as in this paper?

Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully Convolutional Networks for Semantic Segmentation."

1

4 Answers 4

34

For each class Intersection over Union (IU) score is:

true positive / (true positive + false positive + false negative)

The mean IU is simply the average over all classes.


Regarding the notation in the paper:

  • n_cl : the number of classes
  • t_i : the total number of pixels in class i
  • n_ij : the number of pixels of class i predicted to belong to class j. So for class i:

    • n_ii : the number of correctly classified pixels (true positives)
    • n_ij : the number of pixels wrongly classified (false positives)
    • n_ji : the number of pixels wrongly not classifed (false negatives)

You can find the matlab code to compute this directly in the Pascak DevKit here

5
  • @stochastic_zeitgeist check this
    – Miki
    Feb 11, 2017 at 11:52
  • 3
    I think this is only for rectangular bounding boxes. For irregular shapes, I think this should work. Feb 11, 2017 at 14:03
  • 2
    What if I compute IOU like true positive / (true positive + false positive + false negative) for each image in my dataset, what to do next? Should I compute average of all image IOU? What if on some images some classes are not presented and not predicted so I = 0 and U = 0, so do I need to make 'counter' for each class separately when averaging IOU?
    – mrgloom
    Apr 5, 2018 at 14:47
  • 1
    you have to calculate tp/(tp + fp + fn) over all images in your test set. That means you sum up tp, fp, fn over all images in your test set for each class and after that you do calculate the IoU. Taking the average of each individual image IoU results in a wrong global IoU. Aug 20, 2018 at 13:17
  • "Taking the average of each individual image IoU" I'm not saying this. You compute the IoU for all pixels in each class, then you take the average IoU over all classes.
    – Miki
    Aug 20, 2018 at 13:28
16
 from sklearn.metrics import confusion_matrix  
 import numpy as np

 def compute_iou(y_pred, y_true):
     # ytrue, ypred is a flatten vector
     y_pred = y_pred.flatten()
     y_true = y_true.flatten()
     current = confusion_matrix(y_true, y_pred, labels=[0, 1])
     # compute mean iou
     intersection = np.diag(current)
     ground_truth_set = current.sum(axis=1)
     predicted_set = current.sum(axis=0)
     union = ground_truth_set + predicted_set - intersection
     IoU = intersection / union.astype(np.float32)
     return np.mean(IoU)
1
  • I had to use return np.nanmean here, because 0 can be in a valid confusion matrix (us correctly never trying to predict something, leaves 0 confusion) Aug 25, 2020 at 2:23
2

This should help

def computeIoU(y_pred_batch, y_true_batch):
    return np.mean(np.asarray([pixelAccuracy(y_pred_batch[i], y_true_batch[i]) for i in range(len(y_true_batch))])) 

def pixelAccuracy(y_pred, y_true):
    y_pred = np.argmax(np.reshape(y_pred,[N_CLASSES_PASCAL,img_rows,img_cols]),axis=0)
    y_true = np.argmax(np.reshape(y_true,[N_CLASSES_PASCAL,img_rows,img_cols]),axis=0)
    y_pred = y_pred * (y_true>0)

    return 1.0 * np.sum((y_pred==y_true)*(y_true>0)) /  np.sum(y_true>0)
1
  • 1
    The output y_pred is of the shape n_classes*h*w we want to convert these into a single image where the pixel values equals the class at that pixel and that is why we take arg max across the class axis. May 17, 2017 at 15:46
0

The jaccard_similarity_score (per How to find IoU from segmentation masks?) can be used to get the same results as @Alex-zhai's code above:

import numpy as np
from sklearn.metrics import jaccard_score

y_true = np.array([[0, 1, 1],
                   [1, 1, 0]])
y_pred = np.array([[1, 1, 1],
                   [1, 0, 0]])

labels = [0, 1]
jaccards = []
for label in labels:
    jaccard = jaccard_score(y_pred.flatten(),y_true.flatten(), pos_label=label)
    jaccards.append(jaccard)
print(f'avg={np.mean(jaccards)}')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.