4

I have a project which uses BerkelyDB as a key value store for up to hundreds of millions of small records.

The way it's used is all the values are inserted into the database, and then they are iterated over using both sequential and random access, all from a single thread.

With BerkeleyDB, I can create in-memory databases that are "never intended to be preserved on disk". If the database is small enough to fit in the BerkeleyDB cache, it will never be written to disk. If it is bigger than the cache, then a temporary file will be created to hold the overflow. This option can speed things up significantly, as it prevents my application from writing gigabytes of dead data to disk when closing the database.

I have found that the BerkeleyDB write performance is too poor, even on an SSD, so I would like to switch to LMDB. However, based on the documentation, it doesn't seem like there is an option creating a non-persistent database.

What configuration/combination of options should I use to get the best performance out of LMDB if I don't care about persistence or concurrent access at all? i.e. to make it act like an "in-memory database" with temporary backing disk storage?

2 Answers 2

2

Just use MDB_NOSYNC and never call mdb_env_sync() yourself. You could also use MDB_WRITEMAP in addition. The OS will still eventually flush dirty pages to disk; you can play with /proc/sys/vm/dirty_ratio etc. to control that behavior.

5
  • I think this would still cause everything to be written to disk when the database is closed? For a database entirely in memory, this slows down the entire operation significantly. Mar 23, 2016 at 19:41
  • Nonsense. Closing a file doesn't flush the file. If you close and delete the files before the OS flushes, then nothing gets written.
    – hyc
    Mar 23, 2016 at 20:58
  • Just for sake of completeness, you're correct for most (if not all) situations on Linux. But this is not necessarily the case for other operating systems, or even for all filesystems. Tthere is at least some evidence that close() blocks under certain situations on Mac OS X. Source: blog.libtorrent.org/2012/10/asynchronous-disk-io As with all these kinds of tricks, you need to experiment to see if it'll actually work for you. Mar 28, 2016 at 15:15
  • Your reference cites that close() will block when closing a sparse file on HFS+, as the kernel fills in all the empty spaces. That only means that it's filling in logically, in its disk allocation map, there's no mention of whether the changes are written synchronously. POSIX close() doesn't guarantee a flush, which is why fsync() exists. Just because MacOSX close() blocks doesn't mean that it's syncing.
    – hyc
    Mar 31, 2016 at 11:28
  • From the docs, it seems that doing this carelessly might corrupt the database. Is there a certain configuration that won't corrupt the database and just remove Durability from ACID? Aug 28, 2021 at 11:02
0

From this post: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk.

If the dirty ratio is too small, then you will see frequent synchronous disk writes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.