43

We have this use case where we would like to compress and store objects (in-memory) and decompress them as and when required.

The data we want to compress is quite varied, from float vectors to strings to dates.

Can someone suggest any good compression technique to do this ?

We are looking at ease of compression and speed of decompression as the most important factors.

Thanks.

4
  • 3
    java.util.Date occupies no more than a few words of memory. Are you sure compressing it is beneficial?
    – Paul Cager
    May 9, 2011 at 8:49
  • 2
    One question you should ask yourself - Why compress these objects in memory rather than write them to disk (e.g. memory mapped file)? The filesystem can simply be viewed as an extension of the memory hierarchy (i.e. L1 cache, L2 cache, RAM, Disk) with slower access times. You could consider caching recently used objects in memory whilst flushing the remainder to disk. Libraries like Berkeley DB offer this functionality.
    – Adamski
    May 9, 2011 at 9:06
  • Writing them to disk isnt an option as we are very concerned about performance(hence keeping them all in memory).
    – Kichu
    May 10, 2011 at 3:44
  • @Paul We need to compress them since the number of date objects run into millions(but we could live without that as well...yes..you have a point)
    – Kichu
    May 10, 2011 at 3:45

8 Answers 8

67

If you want to compress instances of MyObject you could have it implement Serializable and then stream the objects into a compressed byte array, like so:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gzipOut = new GZIPOutputStream(baos);
ObjectOutputStream objectOut = new ObjectOutputStream(gzipOut);
objectOut.writeObject(myObj1);
objectOut.writeObject(myObj2);
objectOut.close();
byte[] bytes = baos.toByteArray();

Then to uncompress your byte[] back into the objects:

ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
GZIPInputStream gzipIn = new GZIPInputStream(bais);
ObjectInputStream objectIn = new ObjectInputStream(gzipIn);
MyObject myObj1 = (MyObject) objectIn.readObject();
MyObject myObj2 = (MyObject) objectIn.readObject();
objectIn.close();
3
  • 4
    Not every class can implement Serializable, especially if it's a JDK class Jan 4, 2017 at 18:39
  • 1
    There are some good serializing libraries out there that can take care of this issue
    – piegames
    Aug 12, 2017 at 22:25
  • 3
    don't forget gzipOut.finish()
    – Mugen
    Apr 2, 2020 at 5:56
11

Similar to previous answers, except I suggest you use DeflatorOutputStream and InflatorInputStream as these are simpler/faster/smaller than the alternatives. The reason it is smaller is it just does the compression whereas the alternatives add file format extensions like CRC checks and headers.

If size is important, you might like to have a simple serialization of your own. This is because ObjectOutputStream has a significant overhead making small objects much larger. (It improves for larger object especially when compressed)

e.g. an Integer takes 81 byte, and compression won't help much for such a small number of bytes. It is possible to cut this significantly.

2
  • Its not that fast as Gzip for some reason
    – Harish
    Mar 2, 2016 at 5:06
  • 1
    @Harish GZIP is the deflator with a header and a check sum, it has to do more work so it would be surprising if it were slower. Mar 2, 2016 at 9:17
8

One proposal could be to use a combination of the following streams:

1
  • Regarding the compression type are there possibility to adapt org.apache.commons.compress library to compress/decompress java Object serialized/deserialized ?
    – Stefano
    Dec 22, 2021 at 18:57
5

Compression of searilized objects in Java is usually not well... not so good.

First of all you need to understand that a Java object has a lot of additional information not needed. If you have millions of objects you have this overhead millions of times.

As an example lets us a double linked list. Each element has a previous and a next pointer + you store a long value (timestamp) + byte for the kind of interaction and two integers for the user ids. Since we use pointer compression we are 6Bytes * 2 + 8 + 4 * 2= 28Bytes. Java adds 8 Bytes + 12bytes for the padding. This makes 48Bytes per Element.

Now we create 10 million lists with 20 Elements each (time series of click events of users for the last three years (we want to find patterns)).

So we have 200Million * 48 Bytes of elements = 10GB memory (ok not much).

Ok beside the Garbage collection kills us and the overhead inside the JDK skyrocks, we end with 10GB memory.

Now lets use our own memory / object storage. We store it as a column wise data table where each object is actually a single row. So we have 200Million rows in a timestamp, previous, next, userIdA and userIdB collection.

Previous and next are now point to row ids and become 4byte (or 5bytes if we exceed 4billion entries (unlikely)).

So we have 8 + 4 + 4 + 4 + 4 => 24 * 200 Mio = 4.8GB + no GC problem.

Since the timestamp column stores the timestamps in a min max fashion and our timestamps all are within three years, we only need 5bytes to store each of the timestamps. Since the pointer are now stored relative (+ and -) and due the click series are timely closely related we only need 2bytes in average for both previous and next and for the user ids we use a dictionary since the click series are for roughly 500k users we only need three bytes each.

So we now have 5 + 2 + 2 + 3 + 3 => 15 * 200Mio => 3GB + Dictionary of 4 * 500k * 4 = 8MB = 3GB + 8MB. Sounds different to 10GB right?

But we are not finished yet. Since we now have no objects but rows and datas, we store each series as a table row and use special columns being collections of array that actually are storing 5 values and a pointer to the next five values + a pointer previous.

So we have 10Mio lists with 20 enries each (since we have overhead), we have per list 20 * (5 + 3 + 3) + 4 * 6 (lets add some overhead of partly filled elements) => 20 * 11 + 5 * 6 => 250 * 10Mio => 2,5GB + we can access the arrays faster than walking elements.

But hey its not over yet... the timestamps are now relatively stored only requiring 3 bytes per entry + 5 at the first entry. -> so we save a lot more 20 * 9 + 2 + 5 * 6 => 212 * 10Mio => 2,12 GB. And now storing it all to memory using gzip it and we result in 1GB since we can store it all lineary first storing the length of the array, all timestamps, all user ids making it very highly that there are patterns in the bits to be compressable. Since we use a dictionary we just sort it according the propability of each userId to be part of a series.

And since everything is a table you can deserialize everything in almost read speed so 1GB on a modern SSD cost 2 second to load. Try this with serialization / deserialization and you can hear inner user cry.

So before you ever compress serialized data, store it in tables, check each column / property if it can be logically be compressed. And finally have fun with it.

And remember 1TB (ECC) cost 10k today. Its nothing. And 1TB SSD 340 Euro. So do not waste your time on that issue unless you really have to.

3
  • Err, well..first of all, thank you for the long detailed explanation. Our(or rather now their. I have left the job :) ) app is completely in-memory, it has to be and its incredibly complicated and I might have oversimplified the problem. Long story short, The different requirements meant it absolutely had to remain in-memory. We couldnt do anything else.
    – Kichu
    Feb 23, 2015 at 12:34
  • So again, thanks for the detailed explanation. Even though I left the job etc it was good to read that. I think most other answers fundamentally agree that it might be best to use a DB and an SSD would surely help, but alas, we had really stringent performance requirements and we ended up not doing anything. Just bought a few more servers so we could deal with more data (it was a distributed application).
    – Kichu
    Feb 23, 2015 at 12:37
  • I know we do the same. We are going to throw so many hardware at it. What I wrote mostly about was a compact object less memory management braught as 5:1 to 10:1 in actuall memory usage. With such a compact statement we just dump it to disk and have an incredible fast startup without us to rebuild the internal representation. That came finally for free. The data are all in memory no disk access due usage. Feb 23, 2015 at 13:30
2

The best compression technology I know is ZIP. Java supports ZipStream. All you need is to serialize your object into byte array and then zip it.

Tips: Use ByteArrayOutputStream, DataStream, ZipOutputStream.

2
  • 1
    In don't agree. The zip DEFLATE algorithm is quite old and the algorithm used in gzip, bzip2 or 7zip/lzma are probably more efficient.
    – gabuzo
    May 9, 2011 at 8:59
  • 5
    It's pretty useless to have an argument about "best compression technology" without defining whether you're talking about "best speed" or "best compression ratio" - because you can't really have both. May 9, 2011 at 20:38
2

There are various compression algorithm implemented in the JDK. Check the [java.util.zip](http://download.oracle.com/javase/6/docs/api/java/util/zip/package-summary.html) for all algorithm implemented. However it may not be a good thing to compress all your data. For instance a serialized empty array may be several dozen of bytes long as the name of the underlying class is in the serialized data stream. Also most compression algorithm are designed to remove redundancy from large data blocks. On small to medium Java objects you'll probably have very little or no gain at all.

2

This is a tricky problem:

First, using ObjectOutputStream is probably not the answer. The stream format includes a lot of type-related metadata. If you are serializing small objects, the mandatory metadata will make it hard for the compression algorithm to "break even", even if you implement custom serialization methods.

Using DataOutputStream with minimal (or no) added type information will give a better result, but mixed data is not generally that compressible using a general purpose compression algorithms.

For better compression, you may need to look at the properties of the data that you are compressing. For instance:

  • Date objects could be represented as int values if you know that have a precision of 1 day.
  • Sequences of int values could be run-length encoded, or delta-encoded if they have the right properties.
  • and so on.

However way you do it, you will need to do a serious amount of work to get a worthwhile amount of compression. IMO, a better idea would be to write the objects to a database, datastore or file and use caching to keep frequently used objects in memory.

1

If you need to compress arbitrary objects, a possible approach is to serialize the object into a byte array, and then use e.g. the DEFLATE algorithm (the one used by GZIP) to compress it. When you need the object, you can decompress and deserialize it. Not sure about how efficient this would be, but it will be completely general.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.