Background
We maintain a cluster where we have filebeat, metricbeat, packetbeat, etc. shippers pushing data into the cluster. Invariably some index would become hot and we'd want to either disable writing to it for a time or do clean up and reenable indices which had breached their low watermark thresholds and had automatically gone into read_only_allow_delete: true
.
Bash Functions
To ease the management of our clusters for the rest of my team I wrote the following Bash functions to help perform these tasks without having to fumble around with curl
or through Kibana's UI.
$ cat es_funcs.bash
### es wrapper cmd inventory
declare -A escmd
escmd[l]="./esl"
escmd[p]="./esp"
### es data node naming conventions
nodeBaseName="rdu-es-data-0"
declare -A esnode
esnode[l]="lab-${nodeBaseName}"
esnode[p]="${nodeBaseName}"
usage_chk1 () {
# usage msg for cmds w/ 1 arg
local env="$1"
[[ $env =~ [lp] ]] && return 0 || \
printf "\nUSAGE: ${FUNCNAME[1]} [l|p]\n\n" && return 1
}
enable_readonly_idxs () {
# set read_only_allow_delete flag
local env="$1"
usage_chk1 "$env" || return 1
DISALLOWDEL=$(cat <<-EOM
{
"index": {
"blocks": {
"read_only_allow_delete": "true"
}
}
}
EOM
)
${escmd[$env]} PUT '_all/_settings' -d "$DISALLOWDEL"
}
disable_readonly_idxs () {
# clear read_only_allow_delete flag
local env="$1"
usage_chk1 "$env" || return 1
ALLOWDEL=$(cat <<-EOM
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
EOM
)
${escmd[$env]} PUT '_all/_settings' -d "$ALLOWDEL"
}
Example Run
The above functions can be sourced in your shell like so:
$ . es_funcs.bash
NOTE: The arrays at the top of the file map short names for clusters if you happen to have multiple. We have 2, one for our lab and one for our production. So I represented those as l
and p
.
You can then run them like this to enable the read_only_allow_delete
attribute (true) on your l
cluster:
$ enable_readonly_idxs l
{"acknowledged":true}
or p
:
$ enable_readonly_idxs p
{"acknowledged":true}
Helper Script Overview
There's one additional script that contains the curl
commands which I use to interact with the clusters. This script is referenced in the escmd
array at the top of the es_func.bash
file. The array contains names of symlinks to a single shell script, escli.bash
. The links are called esl
and esp
.
$ ll
-rw-r--r-- 1 smingolelli staff 9035 Apr 10 23:38 es_funcs.bash
-rwxr-xr-x 1 smingolelli staff 1626 Apr 10 23:02 escli.bash
-rw-r--r-- 1 smingolelli staff 338 Apr 5 00:27 escli.conf
lrwxr-xr-x 1 smingolelli staff 10 Jan 23 08:12 esl -> escli.bash
lrwxr-xr-x 1 smingolelli staff 10 Jan 23 08:12 esp -> escli.bash
The escli.bash
script:
$ cat escli.bash
#!/bin/bash
#------------------------------------------------
# Detect how we were called [l|p]
#------------------------------------------------
[[ $(basename $0) == "esl" ]] && env="lab1" || env="rdu1"
#------------------------------------------------
# source escli.conf variables
#------------------------------------------------
# g* tools via brew install coreutils
[ $(uname) == "Darwin" ] && readlink=greadlink || readlink=readlink
. $(dirname $($readlink -f $0))/escli.conf
usage () {
cat <<-EOF
USAGE: $0 [HEAD|GET|PUT|POST] '...ES REST CALL...'
EXAMPLES:
$0 GET '_cat/shards?pretty'
$0 GET '_cat/indices?pretty&v&human'
$0 GET '_cat'
$0 GET ''
$0 PUT '_all/_settings' -d "\$DATA"
$0 POST '_cluster/reroute' -d "\$DATA"
EOF
exit 1
}
[ "$1" == "" ] && usage
#------------------------------------------------
# ...ways to call curl.....
#------------------------------------------------
if [ "${1}" == "HEAD" ]; then
curl -I -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
"${esBaseUrl}/$2"
elif [ "${1}" == "PUT" ]; then
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
elif [ "${1}" == "POST" ]; then
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 -H "${contType}" "${esBaseUrl}/$2" "$3" "$4"
else
curl -skK \
<(cat <<<"user = \"$( ${usernameCmd} ):$( ${passwordCmd} )\"") \
-X$1 "${esBaseUrl}/$2" "$3" "$4" "$5"
fi
This script takes a single property file, escli.conf
. In this file you specify the commands to retrieve your username + password from whereever, I use LastPass for that so retrieve them via lpass
as well as setting the base URL to use for accessing your clusters REST API.
$ cat escli.conf
#################################################
### props used by escli.bash
#################################################
usernameCmd='lpass show --username somedom.com'
passwordCmd='lpass show --password somedom.com'
esBaseUrl="https://es-data-01a.${env}.somdom.com:9200"
contType="Content-Type: application/json"
I've put all this together in a Github repo (linked below) which also includes additional functions beyond the above 2 that I'm showing as examples for this question.
References