6

I'm looking for a key value store that can be used from an EC2 instance.

  • item is just an unstructured string, no indexing required
  • item size up to ~5MB but usually below 10kB
  • lots of writes
  • reading doesn't need to be fast, memcache can be put in front that caches frequently needed reads
  • data is too big to fit into memory
  • Eventual Consistency is fine
  • daemon that can be accessed from multiple machines is required

Ideally something AWS hosted would be perfect but:

  • S3 doesn't fit because of too many writes
  • SimpleDB/DynamoDb don't fit because of item size limits and indexing is not required

As there are a lot of key value stores on the market it's hard to choose the best one. Which one would you recommend?

1
  • @caius.howcroft: what do you mean by that?
    – Niko Sams
    Nov 27, 2012 at 6:43

4 Answers 4

6

I found the perfect solution for my use case: memcachedb

It doesn't do fancy document/indexing, it's just a simple key value store.

I didn't do any performance testing yet though.

Edit:

We dropped memcachedb due to problems with replication. Instead we run now mongodb. Mongodb requires much more disk space, and more resources in general. But the replica sets work very reliable and are easy to set up.

4
  • You can use Couchbase that allows you a very fast access to the Key using memcached protocol. Couchbase let you store any type of content associated to the key. Couchbase 2.0 is a document oriented DB but you can store any type of binary content to. Take a look to this paper that will help you to see some of the key benefits: couchbase.com/memcached
    – Tug Grall
    Nov 23, 2012 at 20:13
  • @TugGrall: I think that won't work for my usecase as the data is too big to fit into memory.
    – Niko Sams
    Nov 23, 2012 at 20:18
  • If you choose a "Couchbase Bucket" it will automatically store the content on disk when necessary : couchbase.com/docs/couchbase-manual-1.8/…
    – Tug Grall
    Nov 24, 2012 at 12:07
  • @TugGrall: a that sounds interesting and matches my usecase. Could you create an anwser for that?
    – Niko Sams
    Nov 27, 2012 at 7:00
2

Maybe you should try mongodb:
http://www.mongodb.org/display/DOCS/Amazon+EC2

Quickstart:
http://www.mongodb.org/display/DOCS/Amazon+EC2+Quickstart

Free courses at 10gen and video presentations:
http://www.10gen.com/presentations/nyc-meetup-group/mongodb-and-ec2-a-love-story

Other key-value storages:
http://google-opensource.blogspot.com/2011/07/leveldb-fast-persistent-key-value-store.html

Comments about Riak and their storages especially bitcask and innostore:
http://basho.com/blog/technical/2011/07/01/Leveling-the-Field/

RaptorDB: A extremely small size and fast embedded, noSql, persisted dictionary database using b+tree or MurMur hash indexing. It was primarily designed to store JSON data (see my fastJSON implementation), but can store any type of data that you give it.

HamsterDB: A delightful engine written in C++, which impressed me a lot for its speed while I was using Aarons Watters code for indexing. (RaptorDB eats it alive now... ahem!) It's quite large at 600KB for the 64bit edition.

Esent PersistentDictionary: A project on CodePlex which is part of a another project which implements a managed wrapper over the built in Windows esent data storage engine. The dictionary performance goes down exponentially after 40,000 items indexed and the index file just grows on guid keys. Apparently after talks with the project owners, it's a known issue at the moment.

Tokyo/Kyoto Cabinet: A C++ implementation of key store which is very fast. Tokyo cabinet is a b+tree indexer while Kyoto cabinet is a MurMur2 hash indexer.

4aTech Dictionary: This is another article on CodeProject which does the same thing, the commercial version at the web site is huge (450KB) and fails dismally performance wise on guid keys after 50,000 items indexed.

BerkeleyDB: The grand daddy of all database which is owned by Oracle and comes in 3 flavours, C++ key store, Java key store and XML database.

(Quotation source: http://www.codeproject.com/Articles/190504/RaptorDB)

5
  • I've considered mongodb - but it looks over-designed to me: I don't need document storage, indexing, map reduce etc.
    – Niko Sams
    Nov 21, 2012 at 6:48
  • Maybe Redis or sth mentioned here: stackoverflow.com/questions/1733619/writing-a-key-value-store
    – 42n4
    Nov 21, 2012 at 7:45
  • I need a server. Redis doesn't work as my data is too large to be stored in memory.
    – Niko Sams
    Nov 21, 2012 at 7:58
  • Some comments about leveldb and other storages (riak uses it as its per node storage): google-opensource.blogspot.com/2011/07/…
    – 42n4
    Nov 21, 2012 at 15:07
  • MongoDB isn't designed for heavy writes - more for heavy reads & advanced json quering.
    – mrówa
    Nov 27, 2012 at 15:33
2

Seems like a perfect use case for HBase. It gives great write throughput, especially if your insert keys are somewhat random. HBase is not usually advertised as a K/V store, but it should work just fine. The AWS documentation presents some use cases you might want to have a closer look at. The downside is that HBase can do a lot more than just K/V, so it might be more complex (and complicated) than what you need.

1
+150

Couchbase sounds like a good match for you needs. It's a lot like having memcached with disk storage.

Pros:

  • It's a key/value database. You can store whatever binary blob you want. As of version 2.0 it has support for storing your data as json and running some queries and map/reduce on it. But, if you don't need that, using it as key/value works great.

  • Of all the NoSQL databases I've tried, it's the fastest. This may be because your writes are not immediately committed to disk. Instead, you get an acknowledgment once a write is replicated in the cluster. Data is written to disk asynchronously. So, one potential downside is that if all your nodes crashed simultaneously (e.g. your data center loses power), you may lose data. Depending on the application this may or may not be an issue (and if your whole cluster goes down, you probably have bigger problems).

  • In my experience it has been reliable. If a node goes down, the cluster keeps working and it's very easy to do a failover. Adding new nodes is pretty easy too.

  • Data doesn't have to fit in memory. It gets stored on disk and paged in and out as necessary.

  • The admin interface is very, very nice. It has nifty live graphs to monitor the cluster.

  • It's backwards compatible with the memcached protocol. If you already have code that uses memcached, it'd be pretty straightforward to have it use Couchbase instead.

Cons:

  • The product is still somewhat young, so documentation and support tools are somewhat lacking. This can be a bit annoying sometimes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.