22

We all know Spark does the computation in memory. I am just curious on followings.

  1. If I create 10 RDD in my pySpark shell from HDFS, does it mean all these 10 RDDs data will reside on Spark Workers Memory?

  2. If I do not delete RDD, will it be in memory forever?

  3. If my dataset(file) size exceeds available RAM size, where will data to stored?

0

2 Answers 2

26

If I create 10 RDD in my pySpark shell from HDFS, does it mean all these 10 RDD data will reside on Spark Memory?

Yes, All 10 RDDs data will spread in spark worker machines RAM. but not necessary to all machines must have a partition of each RDD. off course RDD will have data in memory only if any action performed on it as it's lazily evaluated.

If I do not delete RDD, will it be in memory forever?

Spark Automatically unpersist the RDD or Dataframe if they are no longer used. In order to know if an RDD or Dataframe is cached, you can get into the Spark UI -- > Storage table and see the Memory details. You can use df.unpersist() or sqlContext.uncacheTable("sparktable") to remove the df or tables from memory. link to read more

If my dataset size exceeds available RAM size, where will data to stored?

If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time, when they're needed. link to read more

If we are saying RDD is already in RAM, meaning it is in memory, what is the need to persist()? --As per comment

To answer your question, when any action triggered on RDD and if that action could not find memory, it can remove uncached/unpersisted RDDs.

In general, we persist RDD which need a lot of computation or/and shuffling (by default spark persist shuffled RDDs to avoid costly network I/O), so that when any action performed on persisted RDD, simply it will perform that action only rather than computing it again from start as per lineage graph, check RDD persistence levels here.

3
  • 1
    Looks better. You should mention that it is not required to keep all data in memory at any time.
    – user6022341
    Nov 22, 2016 at 7:17
  • @mrsrinivas - "Yes, All 10 RDDs data will spread in spark worker machines RAM."(after performing an action) - if this is the case, why do we need to mark an RDD to be persisted using the persist() or cache() methods on it ?
    – Dipankar
    Nov 23, 2016 at 14:03
  • 1
    We persist/cache rdds to avoid performing cpu/memory/io intensive operations/jobs again in next stages.
    – mrsrinivas
    Nov 23, 2016 at 14:24
12

If I create 10 RDD in my Pyspark shell, does it mean all these 10 RDD data will reside on Spark Memory?

Answer: RDD only contains the "lineage graph" (the applied transformations). So, RDD is not data!!! When ever we perform any action on an RDD, all the transformations are applied before the action. So if not explicitly (of course there are some optimisations which cache implicitly) cached, each time an action is performed the whole transformation and action are performed again!!!

E.g - If you create an RDD from HDFS, apply some transformations and perform 2 actions on the transformed RDD, HDFS read and transformations will be executed twice!!!

So, if you want to avoid the re-computation, you have to persist the RDD. For persisting you have the choice of a combination of one or more on HEAP, Off-Heap, Disk.

If I do not delete RDD, will it be in memory for ever?

Answer: Considering RDD is just "lineage graph", it will follow the same scope and lifetime rule of the hosting language. But if you have already persisted the computed result, you could unpersist!!!

If my dataset size exceed available RAM size, where will data to stored?

Answer: Assuming you have actually persisted/cached the RDD in memory, it will be stored in memory. And LRU is used to evict data. Refer for more information on how memory management is done in spark.

3
  • Assuming there is only one HDFS Read, transformation and action to be done and if the dataset to be computed holds three times the size of RAM, how is the data in RDD partitions loaded in memory for Spark computation?
    – Kannan
    Oct 15, 2017 at 7:44
  • A worker have to have buffer to hold single partition's data and and buffer needed to process the data, other wise the worker will crash!!! The partitioning strategy has to take this in to account!!!
    – rakesh
    Oct 16, 2017 at 10:13
  • Thanks Rakesh for the clarification. So, the size of the buffer should be greater than atleast one partition size. Is there any link which gives me more info on this? Thanks once again.
    – Kannan
    Oct 17, 2017 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.