0

We have developed a java application, whose primary objective is to read a file(input file), process it and convert it into set of output files.

(I have given a generic description of our solution, to avoid irrelevant details).

This program works perfectly fine when the input file is of 4 GB, with memory settings of -Xms4096m -Xmx16384m in a 32 GB RAM

Now we need to run our application with the input file of size 130 GB.

We used a linux box with 250GB RAM and with memory setting of -Xms40g -Xmx200g (also tried couple of other variations) to run the application and hit OutOfMemory Exception.

At this stage of our project it's very hard to consider redesigning the code to accommodate hadoop ( or someother large scale data processing framework), also the current hardware configuration which we can afford is 250GB of RAM.

Can you please suggest us ways to avoid OutOfMemory Exceptions, what is the general practise when developing these kind of applications.?

Thanks in advance

6
  • How are you storing this file in memory once it's read in?
    – Dodd10x
    Jan 16, 2014 at 20:00
  • One option, if the other answers don't work for you, is to use a less memory intensive data structure. I've had to roll my own hashtable that used 1/5 the memory of the built in one to manage data to keep the JRE heap under 500mb.
    – Dodd10x
    Jan 16, 2014 at 20:22
  • The file while being read is added to its proper custom data structures and stored in memory. I use hashtables throughout the conversion process. Jan 16, 2014 at 20:40
  • What type of objects are the stored? If they are primitives you are wasting a large amount of memory using Java Hashmaps.
    – Dodd10x
    Jan 16, 2014 at 21:09
  • They are not primitive data types, we are using some complex objects Jan 16, 2014 at 21:15

2 Answers 2

1

The most obvious thing to try is to not keep the whole file in memory (if possible). So you can maybe process it in chunks, and at any moment of time keep just one or a few chunks in memory (and not the whole file).

1
  • I thought about it a lot, but the data is inter-related and it's very hard to divide the file into chunks. This is one thing we are still actively pursuing. I also wanted to explore other options if there are any support at framework level (like some sort of memory swap out/swap in mechanism) Jan 16, 2014 at 20:48
1

Just try keep using as less memory as you can, say, don't keep whole file in memory, dump it on disk.

Say, Hadoop HDFS does that for you, just check you doesn't have any leaks via a good profiler or heap dump analyser.

A custom solution could be to still use plain files, but organising access in page-like manner. E.g. Java has good MappedByteBuffer that let you load a certain section of a file into memory for faster access (it has certain problems until Java 7, caused unpredictable unmapping, but as far as I know it has since been fixed).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.