So I am trying to implement a custom partitioner using Spark
with Java
, and I found a great example of how to do this online, but it is using Scala
, and I cannot for the life of me figure out how it translates properly into Java so I can try to implement it. Can anyone help? Here is the example code I found for it in Scala
:
class DomainNamePartitioner(numParts: Int) extends Partitioner {
override def numPartitions: Int = numParts
override def getPartition(key: Any): Int = {
val domain = new Java.net.URL(key.toString).getHost()
val code = (domain.hashCode % numPartitions)
if (code < 0) {
code + numPartitions // Make it non-negative
} else {
code
}
}
// Java equals method to let Spark compare our Partitioner objects
override def equals(other: Any): Boolean = other match {
case dnp: DomainNamePartitioner =>
dnp.numPartitions == numPartitions
case _ =>
false
}
}
%
is defined as the remainder, which can be negative.