I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is there an easy, seamless way to do that?
10 Answers
Curator would be an ideal match here. You can find the link here - https://github.com/elastic/curator
A command like below should work just fine -
curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days --timestring '%Y-%m-%d'
You can keep in this in the CRON for removing the indices occasionally.
You can find some examples and docs here - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html
-
This was what I was looking for. Do you have documentation on the aplications of curator? Commented Nov 2, 2015 at 9:52
-
9This does NOT work with curator v4 or newer. It requires a config file and an action file, where the curator's action is described. Commented Aug 31, 2016 at 14:25
-
See answer by @sachchit-bansal for a working curator 4.2 example– chrisanCommented Mar 24, 2017 at 14:08
If you are using elasticsearch version 5.x then you need to install the curator version 4.x. You can see the version compatibility and installation steps from the documentation
Once installed. Then just run the command
curator --config path/config_file.yml [--dry-run] path/action_file.yml
Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag
The configuration file config_file.yml is
---
client:
hosts:
- 127.0.0.1
port: 9200
logging:
loglevel: INFO
logfile: "/root/curator/logs/actions.log"
logformat: default
blacklist: ['elasticsearch', 'urllib3']
The action file action_file.yml is
---
actions:
1:
action: delete_indices
description: >-
Delete indices older than 7 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
exclude:
If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like
#!/bin/bash
# Script to delete the log event indices of the elasticsearch weekly
#This will delete the indices of the last 7 days
curator --config /path/config_file.yml /path/action_file.yml
Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly
and your job is done.
NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.
-
2Thanks, this is the current (2017) working version of this answer for curator 4.2 :)– chrisanCommented Mar 24, 2017 at 14:06
-
That´s the way curator is working! The answer of Vineeth Mohan is outdated from curator 4.x on - and that should apply for most elasticsearch installations right now (where 5.x is current). Commented May 24, 2017 at 13:06
I use a bash script, just change the 30 with the # of days you want to keep
#!/bin/bash
# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`
echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=`echo $LINE | awk '{ print $3 }'`
echo "http://127.0.0.1:9200/$TODELETE"
fi
done
echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=`echo $LINE | awk '{ print $3 }'`
/usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
sleep 1
fi
done
else
echo SCRIPT CLOSED BY USER, BYE ...
echo
exit
fi
-
1It actually works perfectly and doesn't require installing extra tools like Curator. Commented Feb 13, 2019 at 7:22
As of elasticsearch 6.6, Index Lifecycle Management comes included with basic (free) versions elasticsearch, and accomplishes what Curator used to, but in a more graceful way.
The steps below are reproduced without permission from Martin Ehrnhöfer's excellent and concise blog post.
Assumptions (heads up to the copy-pasters):
- Your elasticsearch server is accessible at
http://elasticsearch:9200
- You want your indices to be purged after thirty days (
30d
) - Your policy name will be created as
cleanup_policy
- Your filebeat index names begin with
filebeat-
- Your logstash index names begin with
logstash-
1. Create a policy that deletes indices after one month
curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
-H 'Content-Type: application/json' \
-d '{
"policy": {
"phases": {
"hot": {
"actions": {}
},
"delete": {
"min_age": "30d",
"actions": { "delete": {} }
}
}
}
}'
2. Apply this policy to all existing filebeat and logstash indices
curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
-H 'Content-Type: application/json' \
-d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
-H 'Content-Type: application/json' \
-d '{ "lifecycle.name": "cleanup_policy" }'
3. Create a template to apply this policy to new filebeat and logstash indices
curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
-H 'Content-Type: application/json' \
-d '{
"index_patterns": ["filebeat-*", "logstash-*"],
"settings": { "index.lifecycle.name": "cleanup_policy" }
}'
Take a look at Curator, a tool developed specially for this kind of use case.
A sample command, for the documentation:
curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
--timestring '%Y.%m.%d'
you can use curl
curl -X DELETE http://localhost:9200/filebeat-$(date +"%Y.%m.%d" -d "last Month")
this must to add this command to xxx.sh, and you can create crontab. crontab -e
00 00 * * * /etc/elasticsearch/xxx.sh
this cron will running everyday at 12pm and it will remove old log.
-
1
-
Thanks this helped me! in my case, I used like this:
curl -k -u username:password -X DELETE https://localhost:9200/logstash-$(date +"%Y.%m" -d "-1 month").*
I was doing echo to check the dateecho logstash-$(date +"%Y.%m" -d "-4 month").*
Commented Jul 25, 2022 at 7:14
curator_cli delete_indices --filter_list '{"filtertype":"none"}'
will delete all or filter:
--filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'
yanb (yet another bash)
#!/bin/bash
searchIndex=logstash-monitor
elastic_url=localhost
elastic_port=9200
date2stamp () {
date --utc --date "$1" +%s
}
dateDiff (){
case $1 in
-s) sec=1; shift;;
-m) sec=60; shift;;
-h) sec=3600; shift;;
-d) sec=86400; shift;;
*) sec=86400;;
esac
dte1=$(date2stamp $1)
dte2=$(date2stamp $2)
diffSec=$((dte2-dte1))
if ((diffSec < 0)); then abs=-1; else abs=1; fi
echo $((diffSec/sec*abs))
}
for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" | grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{ print $3 }'); do
date=$(echo ${index: -10} | sed 's/\./-/g')
cond=$(date +%Y-%m-%d)
diff=$(dateDiff -d $date $cond)
echo -n "${index} (${diff})"
if [ $diff -gt 1 ]; then
echo " / DELETE"
# curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty"
else
echo ""
fi
done
Curator didn't helped me
Nowadays Curator is giving me an error when running it using the below command:
curator --config config_file.yml action_file.yml
Error:
Error: Elasticsearch version 7.9.1 incompatible with this version of Curator (5.2.0)
Cannot find the compatible version of curator with Elasticsearch 7.9.1 and I cannot just upgrade or downgrade the elasticsearch version. So instead I used the @Alejandro's answer and did it using the script below. I modified the script a little bit
Script Solution
#!/bin/bash
# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v`
# Just add -u <username>:<password> in curl statement if your elastic search is behind the credentials. Also, you can give an additional grep statement to filter out specific indexes
echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=`echo $LINE | awk '{ print $3 }'`
echo "http://127.0.0.1:9200/$TODELETE"
fi
done
echo
echo -n "Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
echo "$ALLLINES" | while read LINE
do
FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
then
TODELETE=`echo -n $LINE | awk '{ print $3 }'`
/usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
sleep 1
fi
done
else
echo SCRIPT CLOSED BY USER, BYE ...
echo
exit
fi
In my case removing old indices was mandatory since I had Upgraded to 7.5 version from 5.X,
So I followed simple step to clear the indices.
rm -rf /var/lib/elasticsearch/nodes/0/indices/*
-
3