8

What is the best way to do a pagination using Elasticsearch? Currently, I am working on an API that uses Elasticsearch in the backend with Python, and my index does not have much data, so by default we are doing the pagination in the frontend using JavaScript (and so far, we have do not have any problems).

I want to know for bigger indexes, what is the best way to handle pagination:

4
  • 2
    have a look here stackoverflow.com/questions/49320599/…
    – Lupanoide
    Commented Nov 29, 2019 at 13:24
  • How many documents do you have? Is it less than 10k? Commented Dec 2, 2019 at 15:12
  • @NikolayVasiliev, In my index I have like 50k, but I'm thinking of large indexes
    – Andrex
    Commented Dec 2, 2019 at 15:31
  • In this case the link that Lupanoide posted should be of help. Commented Dec 2, 2019 at 16:01

2 Answers 2

19

The default way of paginating over search results in Elasticsearch is using from/size parameters. This will, however, work only for the top 10k search results.

In case you need to go above that the way to go is search_after.

In case you need to dump the entire index, and it contains more than 10k documents, use scroll API.

What's the difference?

All of these queries allow to retrieve portions of search results, but they have major differences.

from/size is the cheapest and fastest, it is what Google would use to go for the second, third, etc. search results pages if it used Elasticsearch.

Scroll API is expensive, because it creates a kind of snapshot of the index the moment you create the first query, to make sure by the end of the scroll you will have exactly the data that was present in the index at the start. Doing a scroll request will cost resources, and running many of them in parallel can kill your performance, so proceed with caution.

Search after instead is a half-way between the two:

search_after is not a solution to jump freely to a random page but rather to scroll many queries in parallel. It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher. For this reason the sort order may change during a walk depending on the updates and deletes of your index.

So it will allow you to paginate above 10k, with a cost of some possible inconsistency.

Why the 10k limit?

index.max_result_window is set to 10k as a hard limit to avoid out of memory situations:

index.max_result_window

The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory.

What about sliced scroll?

Sliced scroll is just a faster way of doing a normal scroll: it allows to download the collection of documents in parallel. Slice is just a subset of documents in the scroll query output.

-2
    response_array = []

response = ElkConfigClient.search index: "index_name",  
      body: {
      query: { 
        bool: { 
          must: [
            "search_query"
          ]
        }
      }
    },
      scroll: '1h', 
      size: 1000

    scroll_id = response["_scroll_id"]
    s_id = scroll_id

     #iterate the response
    response["hits"]["hits"].each do |response|
      response_array.push(response)
    end
    
    while (true)        
      next_response = ElkConfigClient.scroll(scroll_id: s_id, scroll: '1h')
      next_scroll_id = next_response["_scroll_id"]
      s_id = next_scroll_id

      break if next_response["hits"]["hits"].length == 0 

      next_response["hits"]["hits"].each do |response|
        response_array.push(response)
      end
      response_array
    end

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.