1

I have the following matrix

a =

     0    10    10     0     0
     0     5     5     0     0
     1     0     0    50    51
     0     0    10   100   100

I compute the Jaccard distances

D = pdist(a,'jaccard');
D =

1.0000    1.0000    0.7500    1.0000    1.0000    1.0000

and finally I put the distances in a matrix

sim = squareform(D)

sim =

     0    1.0000    1.0000    0.7500
1.0000         0    1.0000    1.0000
1.0000    1.0000         0    1.0000
0.7500    1.0000    1.0000         0

The jaccard index is computed as "One minus the Jaccard coefficient, which is the percentage of nonzero coordinates that differ." (http://www.mathworks.it/help/stats/pdist.html)

The distance between row 1 and 4 is correct (0.75), while the distance between row 1 and 2 should be 0 and is, instead, 1. It seems that when the jaccard similarity is 1, matlab doesn't execute the 1-similarity computation. What am I doing wrong?

4
  • by your definition, shouldn't the distance between 1 and 4 actually be 0.25? (i.e. there are 4 non-zero elements, 3 differ or 75% so 1-0.75 = 0.25) In which case it looks like Matlab just doesn't aaply the 1- part of your definition
    – Dan
    Nov 7, 2014 at 11:08
  • Or perhaps Matlab's definition is ambiguous and they meant "One minus the Jaccard coefficient : which is the percentage of nonzero coordinates that differ."
    – Dan
    Nov 7, 2014 at 11:10
  • It's (1-jaccard index). The jaccard index is: intersection of nonzero elements / union of nonzero elements, so the distance between 1 and 4 is 1-0.25 = 0.75, which is correct.
    – Eugenio
    Nov 7, 2014 at 11:13
  • see my comment to the answer, I was wrong
    – Eugenio
    Nov 7, 2014 at 11:51

1 Answer 1

1

MATLAB seems right to me.

All of the non-zero numbers in rows 1 and 2 differ (in row 1 they're all 10, in row 2 they're all 5), so rows 1 and 2 should have a distance of 1.

Three out of four of the non-zero numbers in rows 1 and 4 differ (10:0, 10:10, 0:100, 0:100), so rows 1 and 4 should have a distance of 0.75.

There seems to be a lot of disagreement about what thing is the Jaccard "coefficient", the Jaccard "index", the Jaccard "similarity" and the Jaccard "distance", and which is one minus the other. MATLAB's documentation doesn't help, as it's not obvious, in the sentence you quote, whether "which" refers to (what MATLAB is describing as) the Jaccard coefficient, or to one minus the Jaccard coefficient.

In any case, whether the terminology used by the MATLAB documentation is correct, the function pdist seems to be giving consistent results, and you can always take one minus whatever it outputs if you want something different.

1
  • You are right; the Jaccard index is often used for unary (0/1) matrixes so I was implicitly assuming that anything > 0 was considered 1, which is not the case. I have to convert my data first. I am pretty sure that "Which" refers to one minus the Jaccard coefficient.
    – Eugenio
    Nov 7, 2014 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.