7

We are running a spark streaming job with yarn as resource manager, noticing that these two directories are getting filled up on the data nodes and we are running out of space when we only run for couple of min's

/tmp/hadoop/data/nm-local-dir/filecache

/tmp/hadoop/data/nm-local-dir/filecache

these directories are not getting cleared automatically , from my research found that this property need's to be set, yarn.nodemanager.localizer.cache.cleanup.interval-ms

Even after setting this up ..it's not automatically clearing out any help will be greatly appreciated

<configuration>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.aux-services</name>
    ~       <value>mapreduce_shuffle</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.resourcemanager.hostname</name>
    ~       <value>hdfs-name-node</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.memory-mb</name>
    ~       <value>16384</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.nodemanager.resource.cpu-vcores</name>
    ~       <value>6</value>
    ~   </property>
    ~
    ~   <property>
    ~       <name>yarn.scheduler.maximum-allocation-mb</name>
    ~       <value>16384</value>
    ~   </property>
         <property>
    ~       <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
    ~       <value>3000</value>
    ~   </property>
    ~
    ~   <!-- Needs to be explicitly set as part of a workaround for YARN-367.
    ~      | If changing this property, you must also change the
    ~      | hadoop.tmp.dir property in hdfs-site.xml. This location must always
    ~      | be a subdirectory of the location specified in hadoop.tmp.dir. This
    ~      | affects all versions of Yarn 2.0.0 through 2.7.3+. -->
    ~   <property>
    ~       <name>yarn.nodemanager.local-dirs</name>
    ~       <value>file:///tmp/hadoop/data/nm-local-dir</value>
    ~   </property>
    ~
    ~ </configuration>

2 Answers 2

5

If the primary problem is that you are running out of space then try setting a lower value for the yarn property "yarn.nodemanager.localizer.cache.target-size-mb". By default this is 10240 MB(10GB).

As for automatic clean up not getting triggered, well it may be due to (or at-least related to) this unresolved bug reported on Yarn 2.7.1: https://issues.apache.org/jira/browse/YARN-4540

1

Your cache cleanup interval is good, but as the local directory is in /tmp, it may be filling up very fast, usually /tmp will have less space. My recommendation is change your yarn.nodemanager.local-dirs to any storage disks like /u01

The recommended value for yarn.nodemanager.localizer.cache.cleanup.interval-ms is 600000 or 10 mins

1
  • Thanks for your comment Bruce even if i point to a different directory ..it get's filled up too ...my question is why is the automatic cleaning not kicking off ...if we can solve that directory where cache is pointing will not be a issue... Commented Jan 9, 2017 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.