Well... I'm back to square one. I can't figure this out for the life of me.
I'm getting the following error:
FATAL ERROR: JS Allocation failed - process out of memory
I could enumerate the dozens (yes, dozens) of things I've tried to get to the root of this problem, but really it would be far too much. So here are the key points:
- I can only get it to happen on my production server, and my app is large and complicated, so it is proving very difficult to isolate
- It happens even though heap size & RSS size are both < 200 Mb, which should not be a problem given that the machines (Amazon Cloud, CentOS, m1.large) have 8Gb RAM
My assumption is that (because of the 2nd point), a leak is probably not the cause; rather, it seems like there's probably a SINGLE object that is very large. The following thread backs up this theory:: In Node.js using JSON.stringify results in 'process out of memory' error
What I really need is some way to find out what the state of the memory is at the moment the application crashes, or perhaps a stack trace leading up to the FATAL ERROR.
Based upon my assumption above, a 10-minute-old heap dump is insufficient (since the object would have not resided in memory).
ulimit
settings allowing it to use all that RAM? Do you know what's going into the app that could create a single object so large? Doesn't seem all that likely that the app is using < 200 MB RAM then a single event suddenly exhausts an additional 7.8 GB. – Peter Lyons Nov 29 '12 at 1:33ulimit -m
andulimit -d
in particular ss64.com/bash/ulimit.html. It seems more realistic that the memory usage is spiking to 1 GB than 7 GB. I guess my next line of inquiry would be the largest requests the app is serving. Are clients streaming large uploads? Are there lots of concurrent clients sending in tons of requests? At least you can follow those clues into the codebase and add extra logging instrumentation. – Peter Lyons Nov 29 '12 at 2:32