2

I am working on semantic search system, that stores huge amount of data. The data actually are documents and their indexes. The main problems are how to index document using ontologies and how to store them.

My question is about the second problem. At first, I implemented storing in RDBMS. It works veeery slowly. I consider to use some NoSQL database for this purpose, but have some doubts.

Please note, that simple text search using Lucene is not what i need in the current field.

Let me simplify the store structure. Note, that only inverted indexes are stored. In RDBMS we have tables:
1) Word - words from some dictionary
2) Document - document with metadata and it's content
3) Hit - word's hits in document (all hits separated by '|')

To get result system analyses words in request and calculate doc relevance basing on word's hit info. I have omitted some moments about semantic analyze, it's not important for now.

What do you think about this structure of the word storing?

{
"word": "some_word",
...
"some other metadata from the dictionary"
...
"hits": [
"doc1" : [ "hit_info1", "hit_info2"...]
"doc2" : [ "hit_info1", "hit_info2"...]
]
}

Thanks in advance!

8
  • Have you considered storing them in your file system? The file system actually is a pretty well optimized NoSQL system. Jan 8, 2012 at 11:35
  • There are many inserts, updates and selects to store are being done, I think file system is not the best solution.
    – Bohdan
    Jan 8, 2012 at 11:41
  • Please provide additional information about your data and how this data is queried. How did you implement this in RDBMS? Jan 8, 2012 at 11:42
  • I'm about to simplify the structure to better understanding. In RDBMS there are tables: Word (words from some dict), Hit (wordid,docid, and all word's hits in this doc), Document(doc itself).
    – Bohdan
    Jan 8, 2012 at 11:46
  • Why not? Filesystems are designed for insert, update, select-by-key. And in fact, any DB, NoSQL or RDBMS, at some point goes down to the filesystem... Jan 8, 2012 at 11:47

1 Answer 1

1

First of all, RDBMS is a good choice for highly structured data. The major performance problem with RDBMS is the transaction processing. You try to manage a n:m relation between words and documents. This can't be done in file system. Use an SQL server and follow following hints, then it should be fast enough.

First of all, you should consider an ORM (object relational mapping) framework that supports "generalized batching". For C# and .NET I can recommend "DataObjects.NET". It saves you a lot of work optimizing client/server round trips.

Make your transaction as large as possible. If you have a document with 1000 words, process it in one transaction. Maybe you can process multiple documents in one transaction.

Form your inserts in two batches: (A batch is a brunch of SQL commands send in one peace to the server)

  1. Query all missing words for your document
  2. Insert the document, the missing words, and the relations in one round.

It is absolutely important to do this in a batch. If you perform single statements you will mess up in client/sever round trips.

I have similar data to process and for a large batch (100000 words) this is done in about 0.2-0.5 seconds.

P.S. And consider to disable flushing to disk on transaction end on your SQL server.

5
  • Thank you for your explanations! I have already implemented batch insert and large transactions. Yes, it does 100k inserts in about 1 sec, but it seems to be not efficient to insert new row for every word's hit in the document.
    – Bohdan
    Jan 8, 2012 at 12:32
  • It is quite simple: If you need it you must have it. If you don't need all the words, think about lazy indexing. Index only the words you query. It is obvious that this has drawbacks on huge amount of documents. Jan 8, 2012 at 12:56
  • Another idea: Have you thought about folding your words into some smaller index with a hash function? calculate a simple hash and fold it to 16bit or less. So your index room is as small as you want. Your result set on queries is too big, but this is easy to check. Maybe you get a better compromise between indexing and querying the data. Jan 8, 2012 at 13:01
  • What can you say about JSON structure for NoSQL storing (i have updated my question)?
    – Bohdan
    Jan 8, 2012 at 13:02
  • In fact you're putting the infos to the words. That won't help you much on inserts, because with every new document you have to update your additional information. That is in fact the same complexity as the n:m table with two keys. I think it is even worse because you have to read, update and write this info for each word in the document. in the n:m table it is only an insert for every word. Jan 8, 2012 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.