-2

Lets say i have this dataframe :

myDataFrame.map(r =>
  (
    ( r.getAs[String]("col1"),
      r.getAs[String]("col2"),
      r.getAs[String]("col3"),
      r.getAs[String]("col4")
    ),
    r.getAs[Int]("col5")))
  .groupByKey()
  .map(r => (r._1, r._2.sum))
  .sortBy(r => r._2)
  .map(r => {
    Chart(r._1._1, r._1._2, r._1._3, r._1._4)
  })
  .toDF("col1", "col2", "col3", "col4")

This returns 1000 dataframes I'd like to keep and return only dataframes from index 10 to 100 . Is it possible ?

Thanks

0

2 Answers 2

0

A Dataset isn't really indexed. How would you define the index over multiple partitions? Plus the data can be shuffled at any time and change the "index". If you just want to take a limited number of elements, use the take(n: Int) method.

2
  • my intention was to make that king of filter after the sort. OK for the take but what about a range ? I mean once sorted only between 10 to 100 Sep 21, 2018 at 14:46
  • I don't think there's anything like this.
    – Dici
    Sep 21, 2018 at 14:56
0

If rows are unique, "limit" with "except" can work (no guarantee):

val df = List(1, 2, 3, 4, 5).toDF("sortable")
df.limit(4).except(df.limit(2))
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.