26

Wondering how to achieve pagination in Lucene, as it does not inherently support pagination. I basically need to search for 'top 10 entries' (based on some parameter) then 'next 10 entries' and so on. And at the same time I don't want Lucene to hog memory. Any piece of advice would be appreciated. Thanks in advance.

1

3 Answers 3

28

You will need to apply your own paging mechanism, something similar to that below.

 IList<Document> luceneDocuments = new List<Document>();

 IndexReader indexReader = new IndexReader(directory);
 Searcher searcher = new IndexSearcher(indexReader);

 TopDocs results = searcher.Search("Your Query", null, skipRecords + takeRecords);
 ScoreDoc[] scoreDocs = results.scoreDocs;

 for (int i = skipRecords; i < results.totalHits; i++)
 {
      if (i > (skipRecords + takeRecords) - 1)
      {
           break;
      }

      luceneDocuments.Add(searcher.Doc(scoreDocs[i].doc));
 }

You will find that iterating the scoreDocs array will be lightweight as the data contained within the index is not really used until the searcher.Doc method is called.

Please note that this example was written against a slightly modified version of Lucene.NET 2.3.2, but the basic principal should work against any recent version of Lucene.

3
  • 1
    I agree, results in Lucene are not as heavy as the results when querying a database so you can easily implement custom pagination methods without having to deal with performance issues Jun 8, 2009 at 7:59
  • 1
    The problem here is when you search for large data set with higher pagination number search is getting slower. It is like you search for a thing then omit a part of the search.
    – Ruwantha
    Aug 29, 2012 at 6:16
  • I am indexing log files, what do you think of using a field for line numbers? Jul 26, 2021 at 16:03
14

Another version of loop, continuing with Kane's code snippet;

....................

ScoreDoc[] scoreDocs = results.scoreDocs;
int pageIndex = [User Value];
int pageSize = [Configured Value];

int startIndex = (pageIndex - 1) * pageSize;
int endIndex = pageIndex * pageSize;
endIndex = results.totalHits < endIndex? results.totalHits:endIndex;

for (int i = startIndex ; i < endIndex ; i++)
{
     luceneDocuments.Add(searcher.Doc(scoreDocs[i].doc));
}
8

I use following way to paginate, may be it help someone. If you know a better strategy, specifically from performance view point, please share.

   public TopDocs search(String query, int pageNumber) throws IOException, ParseException {
        Query searchQuery = parser.parse(query);
        TopScoreDocCollector collector = TopScoreDocCollector.create(1000, true);

        int startIndex = (pageNumber - 1) * MyApp.SEARCH_RESULT_PAGE_SIZE;
        searcher.search(searchQuery, collector);

        TopDocs topDocs = collector.topDocs(startIndex, MyApp.SEARCH_RESULT_PAGE_SIZE);
        return topDocs;
    }
1
  • Hello Asif, could you please explain a bit more on that please? I want to get all the result and then paginate through them. Thanks
    – Ashik
    Aug 3, 2018 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.