If you see this error all of a sudden , then it might be due to time drifts of virtual machines.
All virtual machines can be prone to time drift.
System time can drift several minutes on long running clusters if its not synchronized to a known good time source. So, all of your cluster nodes using their own own system Time's can time drift sporadically over time.
Your Hadoop jobs may initially run successfully, because the drift may not be quite noticeable. However, on long running clusters, if one of the worker time drifted too long( when compared to master's time) that it exceeds the 10 minute interval, then the jobs fail because the YARN containers scheduled on this workers will be marked EXPIRED as soon as the AM submits it.
The key part is:
"For any container, if the corresponding NM doesn’t report to the RM
that the container has started running within a configured interval of
time, by default 10 minutes, the container is deemed as dead and is
expired by the RM."
You can learn more about YARN Container allocation here: http://hortonworks.com/blog/apache-hadoop-yarn-resourcemanager/
So, the jobs will work if you increase the yarn.resourcemanager.rm.container-allocation.expiry-interval-ms
in the yarn-site.xml config file.
But that's just a temporary workaround.
To avoid the actual issue , you need to use some synchronization
mechanism like NTP.
NTP is responsible for time sync with global time servers and your Master/worker nodes.
You need to make sure the NTP daemon is up and running on all nodes of the cluster. NTP also should stay "synchronized" (ntpstat
) during the entire lifecycle of the cluster. Some obvious issues that can cause NTP un-synchronized
- Your firewall may be blocking UDP port 123.
- You may be having AD
environment with a different time sync conflicting with NTP.
http://support.ntp.org/bin/view/Support/TroubleshootingNTP