0

I have been using postgresql for full text search, matching a list of articles against documents containing a particular word. The performance for which degraded with a rise in the no. of rows. I had been using postgresql support for full text searches which made the performance faster, but over time resulted in slower searches as the articles increased.

I am just starting to implement with solr for searching. Going thru various resources on the net I came across that it can do much more than searching and give me finer control over my results.

Solr seems to use an inverted index, wouldn't the performance degrade over time if many documents (over 1 million) contain a search term begin queried by the user? Also if I am limiting the results via pagination for the searched term, while calculating the score for the documents, wouldn't it need to load all of the 1 million+ documents first and then limit the results which would dampen the performance with many documents having the same word?

Is there a way to sort the index by the score itself in the first place which would avoid loading of the documents later?

3
  • What did you mean under "it need to load all of the 1 million+ documents first"? It does not load any document during searching or sorting Mar 18, 2015 at 16:50
  • Wouldn't it need to load all of the documents or calculate the scoring which includes factors idf (inverse document frequency), coord factor for a million of them?
    – Akash
    Mar 18, 2015 at 17:00
  • What score would you want to store in the index? For which query? You want to pre-calculate all possible scores for all possible queries and store them? Just with 10 different words, this would result in 3.628.800 different queries with different scores ...
    – cheffe
    Mar 19, 2015 at 12:56

1 Answer 1

0

Lucene has been designed to solve all the problems you mentioned. Apart from inverted index, there is also posting lists, docvalues, separation of indexed and stored value, and so on.

And then you have Solr on top of that to add even more goodies.

And 1 million documents is an introductory level problem for Lucene/Solr. It is being routinely tested on indexing a Wikipedia dump.

If you feel you actually need to understand how it works, rather than just be reassured about this, check books on Lucene, including the old ones. Also check Lucene Javadocs - they often have additional information.

1
  • Thanks, I suppose fieldcache coupled with various other store features would be sufficient to overcome such problems, just so I meant, a single term having a million docs
    – Akash
    Mar 20, 2015 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.