5

I have an index file in hard as FSdirectory which size is about 2GB. I want to load it in ram to have a better search speed. I use the code behind:

Directory dic=new RAMDirectory(FSDirectory.open(new File("file path")), IOContext.DEFAULT);

But I'm not sure if it work. Do anyone know that it work or not? Any other way better than this on?

I'll be glad of any suggestion.Thanks.

4
  • You'll get better perf loading it directly (don't use RAMDir), the os file cache will load the index into mem for you. Don't forget to setup the jvm Xmx accordingly : let some free mem to the os, at least 2G in your case. (e.g. on a 4G system use Xmx2G).
    – nomoa
    Commented Jul 31, 2013 at 11:13
  • @nomoa I didn't understand what you really mean of loading directly. how to do it? notice that I want to use it in a web application run by tomcat.
    – anony
    Commented Jul 31, 2013 at 11:37
  • I mean that you won't have better search speed with RAMDirectory, look at the java doc which states : Warning: This class is not intended to work with huge indexes. You'll have better search speed by tweaking jvm params and os params (swappiness on linux). You should have a look at some lucene perf FAQ.
    – nomoa
    Commented Jul 31, 2013 at 12:43
  • If the index is read-only, you could get perf gain (at least on first queries) by copying the index on a tmpfs/ramfs filesystem (thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux). Assuming that you have enough physical RAM to do that, swapping will kill perf with that config.
    – nomoa
    Commented Jul 31, 2013 at 12:49

1 Answer 1

2

Yes you can construct RAMDirectory like that but it's strongly discouraged in your case. RAMDirectory is not designed to be used with large data set (more than 100MB). It will be very GC intensive, long to construct and consume precious JAVA Heap space.

If you have perf issues please make sure that you follow all the guidelines descibed in this lucene FAQ

Don't forget that to get descent search speed the os needs to cache some part of the index into memory. In order to do that you must let some free mem available to the OS. If you have 16Gb of physical RAM and a 8gb index you should configure the jvm Xmx with 8g. A nice linux tool to check mem is htop, it will print RAM used for process as green (your heap space) and RAM used as filecache as yellow (the index data).

Some developpers perform warm up queries at application startup to prevent the first user from being stuck few seconds while the index gets loaded by os filecache.

Then if you still think you can get better perf you can try some tricks with RAMfs. But I doubt that you can get better performances than a well tuned os and jvm with MMapDirectory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.