40

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is there an easy, seamless way to do that?

10 Answers 10

34

Curator would be an ideal match here. You can find the link here - https://github.com/elastic/curator

A command like below should work just fine -

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days  --timestring '%Y-%m-%d'

You can keep in this in the CRON for removing the indices occasionally.

You can find some examples and docs here - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

3
  • This was what I was looking for. Do you have documentation on the aplications of curator? Commented Nov 2, 2015 at 9:52
  • 9
    This does NOT work with curator v4 or newer. It requires a config file and an action file, where the curator's action is described. Commented Aug 31, 2016 at 14:25
  • See answer by @sachchit-bansal for a working curator 4.2 example
    – chrisan
    Commented Mar 24, 2017 at 14:08
29

If you are using elasticsearch version 5.x then you need to install the curator version 4.x. You can see the version compatibility and installation steps from the documentation

Once installed. Then just run the command

curator --config path/config_file.yml [--dry-run] path/action_file.yml

Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag

The configuration file config_file.yml is

---
client:
  hosts:
   - 127.0.0.1
  port: 9200
logging:
  loglevel: INFO
  logfile: "/root/curator/logs/actions.log"
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

The action file action_file.yml is

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like

#!/bin/bash
# Script to delete the log event indices of the elasticsearch weekly

#This will delete the indices of the last 7 days
curator --config /path/config_file.yml /path/action_file.yml

Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly and your job is done.

NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.

2
  • 2
    Thanks, this is the current (2017) working version of this answer for curator 4.2 :)
    – chrisan
    Commented Mar 24, 2017 at 14:06
  • That´s the way curator is working! The answer of Vineeth Mohan is outdated from curator 4.x on - and that should apply for most elasticsearch installations right now (where 5.x is current).
    – jonashackt
    Commented May 24, 2017 at 13:06
18

I use a bash script, just change the 30 with the # of days you want to keep

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
  do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else 
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi
1
  • 1
    It actually works perfectly and doesn't require installing extra tools like Curator. Commented Feb 13, 2019 at 7:22
17

As of elasticsearch 6.6, Index Lifecycle Management comes included with basic (free) versions elasticsearch, and accomplishes what Curator used to, but in a more graceful way.

The steps below are reproduced without permission from Martin Ehrnhöfer's excellent and concise blog post.

Assumptions (heads up to the copy-pasters):

  • Your elasticsearch server is accessible at http://elasticsearch:9200
  • You want your indices to be purged after thirty days (30d)
  • Your policy name will be created as cleanup_policy
  • Your filebeat index names begin with filebeat-
  • Your logstash index names begin with logstash-

1. Create a policy that deletes indices after one month

curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "policy": {                       
        "phases": {
          "hot": {                      
            "actions": {}
          },
          "delete": {
            "min_age": "30d",           
            "actions": { "delete": {} }
          }
        }
      }
    }'

2. Apply this policy to all existing filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'

3. Create a template to apply this policy to new filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "index_patterns": ["filebeat-*", "logstash-*"],                 
      "settings": { "index.lifecycle.name": "cleanup_policy" }
    }'
7

Take a look at Curator, a tool developed specially for this kind of use case.

A sample command, for the documentation:

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
   --timestring '%Y.%m.%d'
6

you can use curl

 curl -X DELETE http://localhost:9200/filebeat-$(date +"%Y.%m.%d" -d "last Month")

this must to add this command to xxx.sh, and you can create crontab. crontab -e

00 00 * * * /etc/elasticsearch/xxx.sh

this cron will running everyday at 12pm and it will remove old log.

2
  • 1
    this is all you need.
    – visualex
    Commented Aug 4, 2020 at 8:05
  • Thanks this helped me! in my case, I used like this: curl -k -u username:password -X DELETE https://localhost:9200/logstash-$(date +"%Y.%m" -d "-1 month").* I was doing echo to check the date echo logstash-$(date +"%Y.%m" -d "-4 month").* Commented Jul 25, 2022 at 7:14
1
curator_cli delete_indices --filter_list '{"filtertype":"none"}' 

will delete all or filter:

 --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'
0

yanb (yet another bash)

#!/bin/bash
searchIndex=logstash-monitor
elastic_url=localhost
elastic_port=9200

date2stamp () {
    date --utc --date "$1" +%s
}

dateDiff (){
    case $1 in
        -s)   sec=1;      shift;;
        -m)   sec=60;     shift;;
        -h)   sec=3600;   shift;;
        -d)   sec=86400;  shift;;
        *)    sec=86400;;
    esac
    dte1=$(date2stamp $1)
    dte2=$(date2stamp $2)
    diffSec=$((dte2-dte1))
    if ((diffSec < 0)); then abs=-1; else abs=1; fi
    echo $((diffSec/sec*abs))
}

for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" |     grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{     print $3 }'); do
  date=$(echo ${index: -10} | sed 's/\./-/g')
  cond=$(date +%Y-%m-%d)
  diff=$(dateDiff -d $date $cond)
  echo -n "${index} (${diff})"
  if [ $diff -gt 1 ]; then
    echo " / DELETE"
    # curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty"
  else
    echo ""
  fi
done    
0

Curator didn't helped me

Nowadays Curator is giving me an error when running it using the below command:

curator --config config_file.yml action_file.yml

Error:

Error: Elasticsearch version 7.9.1 incompatible with this version of Curator (5.2.0)

Cannot find the compatible version of curator with Elasticsearch 7.9.1 and I cannot just upgrade or downgrade the elasticsearch version. So instead I used the @Alejandro's answer and did it using the script below. I modified the script a little bit

Script Solution

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v`
# Just add -u <username>:<password> in curl statement if your elastic search is behind the credentials. Also, you can give an additional grep statement to filter out specific indexes

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
    do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo -n $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi
-2

In my case removing old indices was mandatory since I had Upgraded to 7.5 version from 5.X,

So I followed simple step to clear the indices.

rm -rf /var/lib/elasticsearch/nodes/0/indices/*
1
  • 3
    But doing anything bypassing official API is bad.
    – ipeacocks
    Commented May 12, 2021 at 11:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.