88

If a new document is indexed to Elasticsearch index then it is available for searching something like 1 second after index operation. However it can be forced to make this document searchable immediately by calling _flush or _refresh operation on index. What is the difference between these two operations - the result seems to be the same for them, document is immediately searchable.

What exactly does each one of these operations?

ES documentation seems to not tackle this problem deeply.

1
  • 2
    short answer is _refresh DOES make document immediately searchable and _flush DOES NOT Feb 17, 2021 at 20:44

3 Answers 3

139

The answer that you got is correct but I think it's worth to elaborate a bit more.

A refresh effectively calls a reopen on the lucene index reader, so that the point in time snapshot of the data that you can search on gets updated. This lucene feature is part of the lucene near real-time api.

An elasticsearch refresh makes your documents available for search, but it doesn't make sure that they are written to disk to a persistent storage, as it doesn't call fsync, thus doesn't guarantee durability. What makes your data durable is a lucene commit, which is way more expensive.

While you can call lucene reopen every second, you cannot do the same with lucene commit.

Through lucene you can then have new documents available for search in near real-time by calling reopen pretty often, but you still need to call commit to ensure data is written to disk and fsynced, thus safe.

Elasticsearch solves this "problem" by adding a transaction log per shard (effectively a lucene index), where write operations that have not been committed yet are stored. The transaction log is fsynced and safe, thus you obtain durability at any point in time, even for documents that have not been committed yet. You can search on documents in near real-time as refresh happens automatically every second, and you can also be sure that if something bad happens the transaction log can be replayed to restore eventually lost documents. The nice thing about the transaction log is that it can be used internally for other things, for instance to provide real-time get by id.

An elasticsearch flush effectively triggers a lucene commit, and empties also the transaction log, since once data is committed on the lucene level, durability can be guaranteed by lucene itself. Flush is exposed as an api too and can be tweaked, although usually that is not necessary. Flush happens automatically depending on how many operations get added to the transaction log, how big they are, and when the last flush happened.

3
  • 8
    I was using scan & scroll to reindex my data from my old index to a new index. Right after it finishes, I call _flush and/or _refresh and then I call _count api to compare the document counts in the old and the new, expecting them to be equal. But it is not. I have to call those APIs in a loop many times (with 1 second pause at the end of each iteration) until elasticsearch finally gets the correct document count. Is there a way to call some API and confirm that the document counts are accurate? Aug 5, 2015 at 2:07
  • p.s. I set refresh_interval to 30s, but the question remains valid. Aug 5, 2015 at 2:27
  • @javanna, Is after a lucene commit the data is stored to disk in the form segments in the shards?
    – Siddu
    Oct 6, 2018 at 2:02
29

A refresh causes a new segment to be written, so it becomes available for search.

A flush causes a Lucene commit to happen. This is a lot more expensive.

For more details, I've written an article that covers some of this: Elasticsearch from the bottom up :)

20
  1. refresh: transform in-memory buffer to in-memory segment which can be searched.
  2. flush: (a) merge small segments to be a big segment (b) fsync the big segment to disk (c) empty translog.

Refresh:

refresh

Flush:

flush

Segment is a part of lucene. Immutable segments make OS page caches always clean.

Translog is a part of Elasticsearch. Translog is aim for durability.

Reference:

  1. Guide to Refresh and Flush Operations in Elasticsearch
  2. Official Doc for make doc persistent
1

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.