2

We wish to make a desktop application that searches a locally packaged text database that will be a few GB in size. We are thinking of using lucene.

So basically the user will search for a few words and the local lucene database will give back a result. However, we want to prevent the user from taking a full text dump of the lucene index as the text database is valuable and proprietary. A web application is not the solution here as the Customer would like for this desktop application to work in areas where the internet is not available.

How do we encrypt lucene's database so that only the client application can access lucene's index and a prying user can't take a full text dump of the index?

One way of doing this, we thought, was if the lucene index could be stored on an encrypted file system within a file (something like truecrypt). So the desktop application would "mount" the file containing the lucene indexes.

And this needs to be cross platform (Linux, Windows)...We would be using Qt or Java to write the desktop application.

Is there an easier/better way to do this?

[This is for a client. Yes, yes, conceptually this is bad thing :-) but this is how they want it. Basically the point is that only the Desktop application should be able to access the lucene index and no one else. Someone pointed that this is essentially DRM. Yeah, it resembles DRM]

1
  • 3
    I didn't mean "this is evil, just like DRM". I meant "this is broken, just like DRM". Jun 30, 2010 at 12:29

6 Answers 6

5

How do we encrypt lucene's database so that only the client application can access lucene's index and a prying user can't take a full text dump of the index?

You don't. The user would have the key and the encrypted data, so they could access everything. You can bury the key in an obfuscated file, but that only adds a slight delay. It certainly will not keep out prying users. You need to rethink.

4
  • Actually the key would come from one of those standard hardware based software protection USB dongles. Jun 30, 2010 at 12:21
  • 2
    @Sidharth: regardless of where the key is stored, the user has access to both the encrypted data and the key, so you've already lost. It is for this very reason that copy-protection and DRM don't work. (For a reason why a USB dongle would be useless, see this) Jun 30, 2010 at 12:32
  • @Sidharth: a HW dongle might help (when it works - it mostly has a serial-to-USB kludge inside, oozing all kinds of happiness and joy), but then the whole application must be cryptographically secure, otherwise a cracker will just find the place where the HW code is checked and either patch it with "allow always" or dump the decryption key - and you're back to square 1. Jun 30, 2010 at 12:33
  • 2
    We want to make it tough but not impossible. So yes, you could do a memory dump and get the decryption key out (whether you are using a software key or a Hardware based dongle). Is there a way to make this tough for hackers ... we don't want to make it impossible. The point is the client is distributing a database of thousands of valuable text files. He wants customers to be able to search them offline. Searching is fine but indiscriminate ripping of data is not. Jun 30, 2010 at 12:50
3

The problem here is that you're trying to both provide the user with data and deny it from em, at the same time. This is basically the DRM problem under a different name - the attacker (user) is in full control of the application's environment (hardware and OS). No security is possible in such situation, only obfuscation and illusion of security.

While you can make it harder for the user to get to the unencrypted data, you can never prevent it - because that would mean breaking your app. Probably the closest thing is to provide a sealed hardware box, but IMHO that would make it unusable.

Note that making a half-assed illusion of security might be sufficient from a legal standpoint (e.g. DMCA's anti-circumvention clauses) - but that's outside SO's scope.

1
2

Technically, there is little you can do. Lucene is written in Java and Java code can always be decompiled or run in a debugger to get the key which you need to store somewhere (probably in the license key which you sell the user).

Your only option is the law (or the contract with the user). The text data is copyrighted, so you can sue the user if they use it in any way that is outside the scope of the license agreement.

Or you can write your own text indexing system.

Or buy a commercial one which meets your needs.

[EDIT] If you want to use an encrypted index, just implement your own FSDirectory. Check the source for SimpleFSDirectory for an example.

3
  • Yes, people could do that... but we want to make it tough for people, not necessarily impossible. Jun 30, 2010 at 12:23
  • 2
    I suggest to make it more simple for people to use your application so they have no incentive to get the data out. Spend a lot of money to make your legitimate users happy instead of making them mad with copy protection (which will only drive them to the next cracker for a version of your app without copy protection and without any hazzle). Jun 30, 2010 at 12:43
  • 4
    Its not our application... its for a client. They want it like this. We can't get into a moral debate with them. Too often questions like this get mixed up with philosophy :-). We want to know how to do it and not if its good for world hunger :-) Jun 30, 2010 at 12:54
1

Why not building an index that contains only the data that user can access and ship that index with the desktop app?

1

True-crypt sounds like a solid plan to me. You can mount volumes and encrypt them in all sorts of crazy overkill ways, and access them just as any other file.

No, it isn't entirely secure, but it should work well enough.

1

One-way hash function.

You don't store the plaintext, you store hashes. When you want to search for a term, you push the term through the function and then search for the hash. If there's a match in the database, return thumbs up.

Are you willing to entertain false positives in order to save space? Bloom filter.

3
  • If the one way hash function is known to all, the database can be copied and will be still useful to anybody else. Please read question again :-) Aug 1, 2011 at 12:12
  • Since the user can search the database, xe is making a partial copy with every search (a.k.a. "search results"); thus the database can be copied in parts anyway, and then stitching them together: "Hey Joe, open this app and search for everything that begins with 'a', store results; search for everything that begins with 'b', store results; bring me the results when you've run out of letters." The data is accessible, thus protection of the actual database is irrelevant. Aug 1, 2011 at 16:36
  • So add a dash of salt and bake a copy for each customer. If my original comment had not been edited, allusions to this being a forty-year-old problem whose solution is demonstrated in your /etc/passwd file would have made this evident. Aug 19, 2011 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.