0

Here is the start statistics from AWS ElasticSearch

Count: 98365255
Size in Bytes: 62.24 GB

After I deleted about 1/3 of the documents

AWS ElasticSearch dashboard shows

Count: 68782759
Size in Bytes: 57.82 GB

I did not see too much free space after I deleted 1/3 of the documents.

So I called _stats api directly, I got the different Size in Bytes.

"docs": {
            "count": 68782759,
            "deleted": 32680935
        },
        "store": {
            "size_in_bytes": 62078477520,
            "throttle_time_in_millis": 0
        },

The current count match the AWS Elasticsearch dashboard: 68782759.

But the size_in_bytes is

62078477520. 

It is almost the same as the size before I do the bulk delete,

62.24 GB.

In simple words,

I assumed I would get large size of the free space after I deleted 1/3 of the documents.

  • AWS dashboard only shows I get 1/10 of space freed after I deleted 1/3 of the documents
  • _stats shows almost I did not get any space freed after I deleted 1/3 of the documents.

Do I have to wait for the size_in_bytes data to be correct later on?

Do I have to do some kind refresh actions to get the free space after I delete the documents?

Thanks!

1 Answer 1

4

I found the answer here.

https://discuss.elastic.co/t/free-disk-space-monitoring-after-deleting-records/146651

Bernt_Rostad's answer,

"Disc space is not automatically freed when you delete documents from an index. The reason for this is that the index segments, the building blocks of shards, are immutable Lucene indices. This means that a document stored in a segment is never physically updated or erased, just marked as deleted if you execute an update or a delete on it in Elasticsearch.

Because of this, Elasticsearch will perform segment merges in an index from time to time, typically when there are very many small segments in the index or the number of documents marked as deleted is a large percentage of the total number of stored documents ("large" may be 20-30%). When a merge takes place, Elasticsearch will read two or more smaller segments and write them to a new larger one. In the process Elasticsearch will skip all those documents marked as deleted, so that once the new segment is complete and the smaller originals removed, you will have saved disc space corresponding to the size of the deleted documents."

There are approaches to release the space now. But in my case, I will just wait for the segment merging to release the space.

Note:

After wait for one day, the disc space released.

"primaries": {
        "docs": {
            "count": 68784759,
            "deleted": 1885108
        },
        "store": {
            "size_in_bytes": 42676386191,
            "throttle_time_in_millis": 0
        },

Reduced from 62078477520 to 42676386191.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.