I have a file which contain UserId,MovieId,and tags.I want to calculate cosine similarity between tags but tags are not only words but also phrases or sentences.so it is not accurate to calculate similarity between words.I want to calculate similarity between these sentences or phrases in tags.
sample of data
UserId | MovieId | tags |
---|---|---|
109367 | 45571 | "comedy and action" |
88312 | 8432 | "slient comedy" |
3844 | 4565 | "Fantastic comedy!" |
6433 | 7677 | "comedy error" |
279 | 5434 | "Romantic comedy" |
I'm trying to find the cosine similarity between each row in tags.
code:
val lines = scala.io.Source.fromFile("file:///usr/local/spark/dataset/algorithm3/comedy").getLines.mkString
val lines2=scala.io.Source.fromFile("file:///usr/local/spark/dataset/algorithm3/funny").getLines().mkString("\n")
import java.nio.file.Files
import java.nio.file.Paths
val result=textCosine(lines,lines2)
println("The cosine similarity score: "+result)
}
/**
* Modular length of the vector
* @param vec
*/
def module(vec:Vector[Double]): Double ={
// math.sqrt( vec.map(x=>x*x).sum )
math.sqrt(vec.map(math.pow(_,2)).sum)
}
/**
* Find the inner product of two vectors
* @param v1
* @param v2
*/
def innerProduct(v1:Vector[Double],v2:Vector[Double]): Double ={
val listBuffer=ListBuffer[Double]()
for(i<- 0 until v1.length; j<- 0 until v2.length;if i==j){
if(i==j){
listBuffer.append( v1(i)*v2(j) )
}
}
listBuffer.sum
}
/**
* Find the cosine of two vectors
* @param v1
* @param v2
*/
def cosvec(v1:Vector[Double],v2:Vector[Double]):Double ={
val cos=innerProduct(v1,v2) / (module(v1)* module(v2))
if (cos <= 1) cos else 1.0
}
def textCosine(lines:String,lines2:String):Double={
val set=mutable.Set[Char]() //Count all words in two sentences
lines.foreach(set +=_)
lines2.foreach(set +=_)
println(set)
val ints1: Vector[Double] = set.toList.sorted.map(ch => {
lines.count(s => s == ch).toDouble
}).toVector
println("===ints1: "+ints1)
val ints2: Vector[Double] = set.toList.sorted.map(ch => {
lines2.count(s => s == ch).toDouble
}).toVector
println("===ints2: "+ints2)
cosvec(ints1,ints2)
}
}
I want output like this
tag | similarity |
---|---|
"comedy and action | 0.534 |
"comedy error" | 0.435 |
"fantastic comedy" | 0.325 |
"slient comedy | 0.466 |
How can I do it in scala?