0

While calling an api called mapPartitionsWithIndex in spark I have used partial function with case which works pretty fine

val numbersRDD = sc.parallelize(numbers)
      numbersRDD.mapPartitionsWithIndex{ case x: (Int,Iterator[Int]) => Iterator((x._1,x._2.size))}

When I explictly declared it as partialfunction like below

val dosomething: PartialFunction[(Int,Iterator[Int]), Iterator[(Int,Any)]] = {
    case x: (Int,Iterator[Int]) => Iterator((x._1,x._2.size))
  }
  numbersRDD.mapPartitionsWithIndex{dosomething}

Complier starts complaning .I am not sure what I am doing wrong .I might be missing something here.

1 Answer 1

1

Because mapPartitionsWithIndex does not accept a Partial Function but a Function. Even though you are providing a partial function (implementation) in your first code, it's not specifically typed as PartialFunction and the compiler will infer it as Function2.

But in the second piece of code, since you defined it as PartialFunction, compiler cannot convert it into Function2 hence the compilation error.

Why do you want to pass a PartialFunction? The method signature of mapPartitionsWithIndex clearly expects a Function and not a PartialFunction.

  def mapPartitionsWithIndex[U: ClassTag](
      f: (Int, Iterator[T]) => Iterator[U],
      preservesPartitioning: Boolean = false): RDD[U]

PartialFunction is ALWAYS a Function1 i.e. it's of type A => B. Filter accepts a Function1 and hence when you pass partial function to filter, it does not give compilation error, since your partial function will also be sth like A => Boolean. But, the method mapPartitionsWithIndex expects Function2 hence the compilation error. This can be tried out in scala REPL like this:

// Notice the type function1 but mapPartitionsWithIndex expects Function2.
scala>   val dosomething: PartialFunction[(Int, Iterator[Int]), Iterator[(Int, Any)]] = {
     |     case x: (Int, Iterator[Int]) => Iterator((x._1, x._2.size))
     |   }
val dosomething: PartialFunction[(Int, Iterator[Int]),Iterator[(Int, Any)]] = <function1>


// Since this is still function1, it's accepted inside filter. Also, this will still give you MatchError but not compilation.
scala>   val dosomething1: PartialFunction[Int, Boolean] = {
     |     case x: Int if x > 10 => false
     |   }
val dosomething1: PartialFunction[Int,Boolean] = <function1>


2
  • Thanks for reply.I was just curious if I can do that I see filter takes arg of function1 and I can pass partial function there not sure why complier did not complain .But Now I can understand why complier complains Upvoting your answer and please share if you know why filter does not have issue with partial function Sep 29, 2022 at 14:11
  • I edited the answer to include the reason. I hope it's helpful now. Sep 30, 2022 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.