12

Suppose I have an RDD of arbitrary objects. I wish to get the 10th (say) row of the RDD. How would I do that? One way is to use rdd.take(n) and then access the nth element is the object, but this approach is slow when n is large.

1

3 Answers 3

15

I don't know how much it is efficient, as it depends on the current and future optimizations in the Spark's engine, but you can try doing the following:

rdd.zipWithIndex.filter(_._2==9).map(_._1).first()

The first function transforms the RDD into a pair (value, idx) with idx going from 0 onwards. The second function takes the element with idx==9 (the 10th). The third function takes the original value. Then the result is returned.

The first function could be pulled up by the execution engine and influence the behavior of the whole processing. Give it a try.

In any case, if n is very large, this method is efficient in that it does not require to collect an array of the first n elements in the driver node.

3
  • 4
    Unfortunately, zipWithIndex requires a full pass over the data to calculate the index offset of each partition. It is still probably your best bet though.
    – Mike Park
    Jan 7, 2015 at 21:04
  • I tried it and it is slow. Ok for maybe getting one row, but getting several rows, one at a time is very slow
    – Jake
    Nov 8, 2017 at 5:03
  • 3
    Why is this so absurdly hard? .first is all its takes for the first row... Serious question.
    – AlleyOOP
    Feb 2, 2019 at 20:02
10

I haven't checked this for the huge data. But it works fine for me.

Lets say n=2, I want to access the 2nd element,

   data.take(2).drop(1)
3
  • 2
    You don't want to do this for a large n values as it will result in getting the first n elements (which is affected by the partitioning...) to the driver code itself... so it can be slow or even impossible to do...
    – masu
    Aug 29, 2016 at 12:22
  • Agree. Is there any efficient way to do so? Aug 30, 2016 at 6:06
  • AFAIK: Nicola Ferraro's answer above contains the best approach we currently have. stackoverflow.com/a/27826498/2846609
    – masu
    Aug 30, 2016 at 8:47
6

RDD.collect() and RDD.take(x) both return a list, which supports indexing. So each time we need an element at position N.We can perform any of following two codes: RDD.collect()[N-1] or RDD.take(N)[N-1] will work fine when we want element at position N.

1
  • 1
    bad idea if you want the billionth row. This brings the entire RDD to the head end, which is almost always the wrong thing to do.
    – vy32
    Jun 1, 2020 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.