1

I currently have a cluster of 3 machines all running Ubuntu and have been configured using Cloudera Manager. Upon viewing the cluster I see that two of the machines have an error "Memory Overcommit Validation Threshold". One is using 6.3GB of memory when it only has 3.8GB of physical memory in the machine, and the other is using 11.6GB of memory when there is only 7.8GB of physical memory in the machine.

I think this is because those two are set both as datanodes and namenodes, but I am not sure how to make it so that it is only using the memory available to it and not swapping. I have already set swappiness to 0 in the sysctl.conf file.

I am not able to use another machine for the namenode or add any memory to the machines. Any idea how I could lower the memory usage, as well as what I should lower to what extent so the cluster doesn't start to fail due to too little memory being available?

1 Answer 1

3

This is completely unrelated to Swappiness. This error occurs only when memory allocated for services on that node is greater than the total memory available for that node (Note that 20% of the Memory is reserved for the system by default)

Go to the Resources page on each host to see how much memory each service is using. You can lower the amount of memory used by each service by going to the configuration tab (for each service) and changing it (Use "memory" or "heap" in the search bar of the configuration pages to bring those properties up)

Doing this would get rid of that error but this is not a recommended fix. The reason for this error in the first place is that you might be having too many services running on your cluster. Try getting rid of some of them that you don't use. You can always add them later (when you add more memory)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.