5

My JVM-based app runs on Amazon ECS. Under some conditions that I cannot reproduce on test environments it crashes with an OutOfMemoryError. I want to capture the JVM heap dump (using HeapDumpOnOutOfMemoryError) to see what's going on. But there is a problem - because my app uses a lot of memory, the heap dumping takes very long time and ECS kills the container in the middle of it, because the health checks fail. I end up with an unusable, truncated dump file.

I can obtain a correct heap dump by changing the health-check configuration but it potentially affects the ECS service, because without good health-check the cluster can be running unresponsive instances.

I'm looking for a way, that will let me capture the heap dumps automatically with smallest possible impact to the production cluster.

2 Answers 2

2

The way I found:

  1. prepare a copy of your ECS service (Service2) identical in everything but:
    • the health-check settings (disable health-checks)
    • the HeapDumpOnOutOfMemoryError JVM option (enable it)
    • the number of nodes (set to 1)
  2. make the Service2 handle the production traffic alongside the existing Service1 (for example by routing HTTP traffic via a load balancer that forwards traffic to both Service1 and Service2)
  3. wait until the node where the ECS Task of Service2 is running produces the JVM heap dump
  4. scale down Service2 to zero nodes
0

Is this around a Fargate instance or EC2?

In either case, you can add an EC2 instance in the same cluster, and the node, even when the task fails, would be able to login to the host and fetch the heap dump.

1
  • 1
    Please read my question carefully and note that 1) I want an automated solution 2) JVM heap dump takes more than ECS wants to wait before killing the heap-dumping process in the middle, leaving the dump broken. So just logging in to the host won't cut it, sadly. Jun 3, 2020 at 20:00

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.