0

This is part of my code.

            idf=self.getInverseDocFre(word)  ##this idf is from the collection
            qi=count*idf
            di=self.docTermCount[docid][word]*idf
            similiarity+=qi*di
            similiarity/=self.docSize[docid]

this is wikipedia https://en.wikipedia.org/wiki/Vector_space_model#Example:_tf-idf_weights

this is an example from web http://www.site.uottawa.ca/~diana/csi4107/cosine_tf_idf_example.pdf

My question is that, if idf for the query is the same idf from the collection?

Is that why I have to multiply the idf for the similiarity twice?

I am afraid that I am wrong about the concept of idf for the query part.

1 Answer 1

1

You have to represent your query in the same space as the documents of your collection, i.e. that transformation of words->vectors has to be the same for both, the documents and the query, otherwise you would be comparing apples to oranges. This transformation is fixed once you have extracted the terms and calculate the IDFs from the collection. Once you have this you can represent new word documents in this representation.

Imagine that your query is exactly one of your documents(d2 for example):

d2 =    [0 0 0.584 1.584 0 0.584]  # new york post
query = [0 0     1     1 0     1]  # new york post

In this case you expect the similarity to be one. There is no way this is going to happen if you don't multiply the query TFs by the corresponding IDFs(which you got from the collection). A vector that has only counts(term frequencies) is not going to be parallel to a vector that has each component multiplied by its corresponding idf (except in the special case that all idf are equal). That is why you have to multiply the query too, because the documents have already been multiplied.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.