15

I would like to provide my system with a way of detecting whether out of memory exception has occurred or not. The aim for this exercise is to expose this flag through JMX and act correspondingly (e.g. by configuring a relevant alert on the monitoring system), as otherwise these errors sit unnoticed for days.

Naive approach for this would be to set an uncaught exception handler for every thread and check whether the raised exception is instance of OutOfMemoryError and set a relevant flag. However, this approach isn't realistic for the following reasons:

  • The exception can occur anywhere, including 3rd party libraries. There is nothing I can do to prevent them catching Throwable and keeping it for themselves.
  • Libraries can spawn their own threads and I have no way of enforcing uncaught exception handlers for these threads.

One of possible scenarios I see is bytecode manipulation (e.g. attaching some sort of aspect on top of OutOfMemoryError), however I am not sure if that's right approach or whether this is doable in general.

We have -XX:+HeapDumpOnOutOfMemoryError enabled, but I don't see this as a solution for this problem as it was designed for something else - and it provides no Java callback when this happens.

Has anyone done this? How would you solve it or suggest solving it? Any ideas are welcome.

6
  • 3
    This may be useful stackoverflow.com/questions/3821144/… You can run a script in the event of an out of memory error. Commented Jul 16, 2012 at 16:19
  • Thanks Peter - I wasn't aware of this SO post. Sadly solution provided there is for shell and not for Java :(
    – mindas
    Commented Jul 16, 2012 at 16:23
  • Short of spawning a process to run your code, and monitoring for the process to fail, there's not really a good way to monitor for out of memory error in Java.
    – Hot Licks
    Commented Jul 16, 2012 at 16:34
  • 3
    Your shell script can do anything including call JMX in your JVM. The problem with the JVM trying to handle an OOME is that this can fail due to an OOME. Commented Jul 16, 2012 at 16:36
  • 1
    Which you can do via a JMX call from a program triggered this way. Commented Jul 16, 2012 at 16:49

4 Answers 4

6

You could use an out of memory warning system; this OutOfMemoryError Warning System can be an inspiration. You could configure a listener which is invoked after a certain memory threshold ( say 80%) is breached - you can use this invocation to start taking corrective measures.

We use something similar, where we suspend the component's service when the memory threshold of the component reaches 80% and start the clean up action; the component comes back only when the used memory comes below a another configurable value threshold.

3

There is an article based on the post that Scorpion has already given a link to.

The technique is again based on using MemoryPoolMXBean and subscribing to the "memory threshold exceeded" event, but it's slightly different from what was described in original post.

Author states that when you subscribe for the plain "memory threshold exceeded" event, there is a possibility of "false alarm". Imagine a situation when the memory consumption is above the threshold, but there will be a garbage collection performed soon and a lot of the memory is freed after that. In fact that situation is quite common in real world applications.

Fortunately, there is another threshold, "collection usage threshold", and a corresponding event, which is fired based on memory consumption right after garbage collection. When you receive that event, you can be much more confident you're running out of memory.

1
  • 1
    Please note that the article suggests to set the threshold as int which is not suitable for thresholds higher than 2 GB (it will silently truncate a greater value). You should cast it to long or just use Math.round() instead of Math.floor() method.
    – Toparvion
    Commented Aug 8, 2022 at 5:19
0

We have -XX:+HeapDumpOnOutOfMemoryError enabled, but I don't see this as a solution for this problem as it was designed for something else - and it provides no Java callback when this happens.

This flag should be all that you need. Set the output directory of the resulting heap dump file in some known location that you check regularly. Having a callback would be of no use to you. If you are out of memory, you can't guarantee that the callback code has enough memory to execute! All you can do is collect the data and use an external program to analyze why you ran out of memory. Any attempt at recovering in process can create bigger problems.

Bytecode instrumentation is possible - but hard. HPjmeter's monitoring tool has the ability to predict future OOM's (with caveats) -- but only on HP-UX/Itanium based systems. You could dedicate a daemon thread to calculating used memory in process and trigger an alert when this is exceeded, but really you're not solving the problem.

3
  • I have explicitly mentioned this solution is inappropriate - we are running multiple instances of our software on different machines and having a single shared nfs folder for all dumps, as they tend to be quite large (4G+). I'd rather know there is no proper solution than invest my time and build something that is crippled
    – mindas
    Commented Jul 16, 2012 at 18:06
  • Why would you have a single NFS point for all the dumps? It sounds like you need to rethink your setup. Tell me, what is a proper solution? What could you do, given that you just ran out of memory?? Commented Jul 16, 2012 at 18:49
  • If you think such a setup is bad, it would be nice of you to explain why. But nevertheless I'll explain my point. We don't want a spurious memory dump to overflow other, business-critical, partitions and interrupt business flow. This is pretty much the same argument on why you need to separate /var/log from /home in Unix. On top of that, we are running dozens of virtual machines and maintaining a single mount point is far more convenient than just looking for files among multiple servers.
    – mindas
    Commented Jul 17, 2012 at 8:44
0

You can catch any and all uncaught exceptions with the static Thread.setDefaultUncaughtExceptionHandler. Of course, it doesn't help if someone is catching all Throwables. (I don't think anything will, though with an OOME I'd suspect you'd get a cascading effect until something outside the offending try block blew up.) Hopefully the thread would have released enough memory for the exception handler to work; OOM errors do tend to multiply as you try to deal with them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.