6

When I explicitly call rdd.cache, I can see from the spark console storage tab that only a fraction of the rdd is actually cached. My question is where are the remaining parts? How does Spark decide which part to leave in cache?

The same question applies to the initial raw data read in by sc.textFile(). I understand these rdd's are automatically cached, even though the spark console storage table does not display any information on their cache status. Do we know how much of those are cached vs. missing?

1 Answer 1

5

cache() is the same as persist(StorageLevel.MEMORY_ONLY), and your amount of data probably exceeds the available memory. Spark then evicts caches in a "least recently used" manner.

You can tweak the reserved memory for caching by setting configuration options. See the Spark Documentation for details and look out for: spark.driver.memory, spark.executor.memory, spark.storage.memoryFraction

Not an expert, but I do not think that textFile() automatically caches anything; the Spark Quick Start explicitly caches a text file RDD: sc.textFile(logFile, 2).cache()

3
  • Thanks for the explanation. I saw some low percent cache number. It did not appear that the missing portion is caused by spark executors running out of memory. Can't be sure. Also, I read once that input data is automatically cached. I can no longer find the article. I will keep looking.
    – bhomass
    Apr 12, 2015 at 17:28
  • Default cache memory is at maximum 300m I assume (512mb default heap size * 0.6 memoryFraction, minus some other overhead). It depends on your setup if you need to increase driver or executor memory. Do you run Spark locally or with a cluster? If locally, you only need to tweak spark.driver.memory.
    – stholzm
    Apr 13, 2015 at 4:37
  • If you are caching because the initial step is expensive to compute and caching as an object in memory is not required , a better tweak would be to use the persit method with an alternative storage level. spark.apache.org/docs/latest/… MEMORY_ONLY_SET , allows you to squeeze more in memory (x4 with my data structure) , the DISK options allow you to cache on disk
    – Stéphane
    Jun 3, 2015 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.