29

Is there any open-source alternative for Terracotta BigMemory?

Actually I didn't even manage to find any commercial alternative. I'm interested in pure Java solution which will work inside JVM without any JNI and C-backed solution.

0

7 Answers 7

19

There is a very good cache solution named MapDB(JDBM4 formerly). It supports HashMap and TreeMap But it is only application embedded. It also support persistent file based cache.

Example for off heap cache:

DB db = DBMaker.newDirectMemoryDB().make();
ConcurrentNavigableMap<Integer, String> map = db.getTreeMap("MyCache");

Or persistent file based cache:

DB db = DBMaker.newFileDB(new File("/home/collection.db")).closeOnJvmShutdown().make();
ConcurrentNavigableMap<Integer,String> map = db.getTreeMap("MyCache");
2
  • 1
    It's disk backed solution. Not really what I'm asking about.
    – Tema
    Nov 13, 2012 at 16:42
  • 1
    Worth mentioning that MapDB was developed in Kotlin and will impose a dependency on its runtime. Oct 18, 2017 at 20:53
11

I've been having this question myself so I'm just going to update the previous answers with my findings.

I found this thread from quora which also talks about the same question:

http://www.quora.com/JVM/Whats-the-best-open-source-solution-for-java-off-heap-cache

The different solution that seem to be a good fit, besides the directmemory (which has not really been updated in the last year) are

  • MapDB - this seems to be a very complete solution that does much more then off-heap caching and supports a lot of features
  • HugeCollections - This seems to be much less complex application then MapDB, which is focused on allocating off-heap data by extending ConcurrentMap and Map. A fork project from this, meant to target Java 8, is Chronicle-Map. A nice article about this is http://blog.shinetech.com/2014/08/26/using-hugecollections-to-manage-big-data/
  • SpyMemcached - this is a very simple single-threaded implementation with good reputation on github.
  • xmemcached - this also has a fair reputation on github but it doesn't seem to be very talked about.
  • Fast serialization - also focused on reimplementing Java Serialization with focus on off-heap usage of memory - http://ruedigermoeller.github.io/fast-serialization/

However, I would be interested furthermore to find a big enough application that is using any of these three: directmemory, SpyMemcached, xmemcached. Should I find one I will update this answer.

9

Looks like there is a proposal at apache:

http://wiki.apache.org/incubator/DirectMemoryProposal

2
  • 4
    The proposal has been accepted. DirectMemory is on the Apache incubator.
    – javanna
    Jan 16, 2012 at 14:59
  • 2
    That project didn't have a long life and has now been retired. Nov 16, 2015 at 9:14
9

I am developing a solution to be much faster, but I wouldn't suggest you use it just yet as it just a proof of concept at this stage.

http://vanillajava.blogspot.com/2011/09/new-contributors-to-hugecollections.html

However if you have a specific requirement, it may be easier to code it yourself, to use direct ByteBuffers or memory mapped files.

e.g.

// using native order speeds access for values longer than a byte.
ByteBuffer bb = ByteBuffer.allocateDirect(1024*1024*1024).order(ByteOrder.nativeOrder());
// start at some location.
bb.position(0);
bb.put((byte) 1);
bb.putInt(myInt);
bb.putDouble(myDouble);

// to read back.
bb.position(0);
byte b = bb.get();
int i = bb.getInt();
double d = bb.getDouble();

You can do similarly for memory mapped files. Memory mapped files don't count towards you direct memory limit and don't use up swap space.

Are you sure BigMemory won't do the job for you?

3
  • 2
    I'd like free of charge tool so BigMemory doesn't feet. It's not so easy to code it myself because I need add and remove data from cache, so I'll have to code some complicated logic similar to GC in an allocated buffer.
    – Tema
    Oct 10, 2011 at 15:38
  • 3
    Fast-forward 6 years, this project is now called Chronicle Map
    – leventov
    Mar 19, 2017 at 0:08
  • @peter-lawrey can Chronicle solve a similar problem like this: stackoverflow.com/questions/63926180/…
    – quarks
    Sep 16, 2020 at 20:42
4

https://github.com/raffaeleguidi/DirectMemory

2
4

Although it isn't a solution, a guide to how to make use of ByteBuffers for your use case has been written about by Keith Gregory. Take a look at http://www.kdgregory.com/programming/java/ByteBuffer_JUG_Presentation.pdf for an overview and http://www.kdgregory.com/index.php?page=java.byteBuffer for the nitty-gritty details.

4

This implementation of a java off-heap cache uses direct memory and provides good performance in a light-weight java library:

https://github.com/snazy/ohc

Take a look at the benchmarking section for performance numbers. It's licensed under Apache 2.

1
  • "This project is no longer maintained" Apr 11 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.