In my Redis DB I have a number of prefix:<numeric_id>
hashes.
Sometimes I want to purge them all automatically. How do I do this without using some distributed locking mechanism?
In my Redis DB I have a number of prefix:<numeric_id>
hashes.
Sometimes I want to purge them all automatically. How do I do this without using some distributed locking mechanism?
Execute in bash:
redis-cli KEYS "prefix:*" | xargs redis-cli DEL
UPDATE
Ok, i understood. What about this way: store current additional incremental prefix and add it to all your keys. For example:
You have values like this:
prefix_prefix_actuall = 2
prefix:2:1 = 4
prefix:2:2 = 10
When you need to purge data, you change prefix_actuall first (for example set prefix_prefix_actuall = 3), so your application will write new data to keys prefix:3:1 and prefix:3:2. Then you can safely take old values from prefix:2:1 and prefix:2:2 and purge old keys.
redis-cli KEYS "prefix:*" | xargs --delim='\n' redis-cli DEL
Commented
Sep 16, 2011 at 13:31
redis-cli -n 3 KEYS "prefix:*" | xargs redis-cli -n 3 DEL
Commented
May 19, 2013 at 18:08
Starting with redis 2.6.0, you can run lua scripts, which execute atomically. I have never written one, but I think it would look something like this
EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 prefix:[YOUR_PREFIX e.g delete_me_*]
Warning: As the Redis document says, because of performance maters,
keys
command should not use for regular operations in production, this command is intended for debugging and special operations. read more
See the EVAL documentation.
EVAL "local keys = redis.call('keys', ARGV[1]) \n for i=1,#keys,5000 do \n redis.call('del', unpack(keys, i, math.min(i+4999, #keys))) \n end \n return keys" 0 prefix:*
del prefix:*
should be a fundamental operation :/
EVAL "return redis.call('del', 'defaultKey', unpack(redis.call('keys', ARGV[1])))" 0 prefix:*
Commented
Dec 1, 2016 at 15:27
Here's a completely working and atomic version of a wildcard delete implemented in Lua. It'll run much faster than the xargs version due to much less network back-and-forth, and it's completely atomic, blocking any other requests against redis until it finishes. If you want to atomically delete keys on Redis 2.6.0 or greater, this is definitely the way to go:
redis-cli -n [some_db] -h [some_host_name] EVAL "return redis.call('DEL', unpack(redis.call('KEYS', ARGV[1] .. '*')))" 0 prefix:
This is a working version of @mcdizzle's idea in his answer to this question. Credit for the idea 100% goes to him.
EDIT: Per Kikito's comment below, if you have more keys to delete than free memory in your Redis server, you'll run into the "too many elements to unpack" error. In that case, do:
for _,k in ipairs(redis.call('keys', ARGV[1])) do
redis.call('del', k)
end
As Kikito suggested.
for _,k in ipairs(redis.call('keys', KEYS[1])) do redis.call('del', k) end
unpack
transforms a table in a "list of independent variables" (other languages call that explode
) but the max number is not dependent on the syste memory; it's fixed in lua through the LUAI_MAXSTACK
constant. In Lua 5.1 & LuaJIT it's 8000 and in Lua 5.2 is 100000. The for loop option is recommended IMO.
EVAL
since it doesn't specify in advance the keys that it will operate on. It should work on a single instance but don't expect it to work with Redis Cluster.
Commented
Dec 12, 2015 at 5:17
Disclaimer: the following solution doesn't provide atomicity.
Starting with v2.8 you really want to use the SCAN command instead of KEYS[1]. The following Bash script demonstrates deletion of keys by pattern:
#!/bin/bash
if [ $# -ne 3 ]
then
echo "Delete keys from Redis matching a pattern using SCAN & DEL"
echo "Usage: $0 <host> <port> <pattern>"
exit 1
fi
cursor=-1
keys=""
while [ $cursor -ne 0 ]; do
if [ $cursor -eq -1 ]
then
cursor=0
fi
reply=`redis-cli -h $1 -p $2 SCAN $cursor MATCH $3`
cursor=`expr "$reply" : '\([0-9]*[0-9 ]\)'`
keys=${reply##[0-9]*[0-9 ]}
redis-cli -h $1 -p $2 DEL $keys
done
[1] KEYS is a dangerous command that can potentially result in a DoS. The following is a quote from its documentation page:
Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.
UPDATE: a one liner for the same basic effect -
$ redis-cli --scan --pattern "*:foo:bar:*" | xargs -L 100 redis-cli DEL
-n 1
to each redis-cli
invocation: redis-cli -n 1 --scan --pattern "*:foo:bar:*" | xargs -L 100 redis-cli -n 1 DEL
Commented
Mar 20, 2018 at 5:11
For those who were having trouble parsing other answers:
eval "for _,k in ipairs(redis.call('keys','key:*:pattern')) do redis.call('del',k) end" 0
Replace key:*:pattern
with your own pattern and enter this into redis-cli
and you are good to go.
Credit lisco from: http://redis.io/commands/del
I am using below command in redis 3.2.8
redis-cli KEYS *YOUR_KEY_PREFIX* | xargs redis-cli DEL
You can get more help related to keys pattern search from here :- https://redis.io/commands/keys. Use your convenient glob-style pattern as per your requirement like *YOUR_KEY_PREFIX*
or YOUR_KEY_PREFIX??
or any other.
And if any of you have integrated Redis PHP library than below function will help you.
flushRedisMultipleHashKeyUsingPattern("*YOUR_KEY_PATTERN*"); //function call
function flushRedisMultipleHashKeyUsingPattern($pattern='')
{
if($pattern==''){
return true;
}
$redisObj = $this->redis;
$getHashes = $redisObj->keys($pattern);
if(!empty($getHashes)){
$response = call_user_func_array(array(&$redisObj, 'del'), $getHashes); //setting all keys as parameter of "del" function. Using this we can achieve $redisObj->del("key1","key2);
}
}
Thank you :)
You can also use this command to delete the keys:-
Suppose there are many types of keys in your redis like-
Ex- 'xyz_category_fpc' here xyz is a sitename, and these keys are related to products and categories of a E-Commerce site and generated by FPC.
If you use this command as below-
redis-cli --scan --pattern 'key*' | xargs redis-cli del
OR
redis-cli --scan --pattern 'xyz_category_fpc*' | xargs redis-cli del
It deletes all the keys like 'xyz_category_fpc' (delete 1, 2 and 3 keys). For delete other 4, 5 and 6 number keys use 'xyz_product_fpc' in above command.
If you want to Delete Everything in Redis, then follow these Commands-
With redis-cli:
For Example:- in your shell:
redis-cli flushall
redis-cli flushdb
redis-cli del
is not atomic.
Commented
Jan 12, 2017 at 22:13
redis-cli --scan --pattern 'xyz_category_fpc*' | xargs -I{} redis-cli del '{}'
redis-cli --scan --pattern 'xyz_category_fpc*' | redis-cli --pipe del
it improving efficiency and handling keys with spaces or special characters correctly.
Commented
Aug 9 at 9:00
@mcdizle's solution is not working it works only for one entry.
This one works for all keys with same prefix
EVAL "for i, name in ipairs(redis.call('KEYS', ARGV[1])) do redis.call('DEL', name); end" 0 prefix*
Note: You should replace 'prefix' with your key prefix...
I succeeded this with the simplest variant of EVAL command:
EVAL "return redis.call('del', unpack(redis.call('keys', 'my_pattern_here*')))" 0
where I replaced my_pattern_here
with my value.
EVAL "return redis.call('del', unpack(redis.call('keys', 'my_pattern_here*')))" 0
If you have space in the name of the keys, you can use this in bash:
redis-cli keys "pattern: *" | xargs -L1 -I '$' echo '"$"' | xargs redis-cli del
// TODO
You think it's command not make sense bu some times Redis command like DEL
not working correct and comes to the rescue of this
redis-cli KEYS "*" | xargs -i redis-cli EXPIRE {} 1
it's life hack
Other answers may not work if your key contains special chars - Guide$CLASSMETADATA][1]
for instance. Wrapping each key into quotes will ensure they get properly deleted:
redis-cli --scan --pattern sf_* | awk '{print $1}' | sed "s/^/'/;s/$/'/" | xargs redis-cli del
this is the easiest way that comes to mind without using any xargs magic
pure bash!
redis-cli DEL $(redis-cli KEYS *pattern*)
redis-cli -h localhost -n 1 DEL $(redis-cli -h localhost -n 1 KEYS *pattern*)
. Thank you.
@itamar's answer is great, but the parsing of the reply wasn't working for me, esp. in the case where there are no keys found in a given scan. A possibly simpler solution, directly from the console:
redis-cli -h HOST -p PORT --scan --pattern "prefix:*" | xargs -n 100 redis-cli DEL
This also uses SCAN, which is preferable to KEYS in production, but is not atomic.
I just had the same problem. I stored session data for a user in the format:
session:sessionid:key-x - value of x
session:sessionid:key-y - value of y
session:sessionid:key-z - value of z
So, each entry was a seperate key-value pair. When the session is destroyed, I wanted to remove all session data by deleting keys with the pattern session:sessionid:*
- but redis does not have such a function.
What I did: store the session data within a hash. I just create a hash with the hash id of session:sessionid
and then I push key-x
, key-y
, key-z
in that hash (order did not matter to me) and if I dont need that hash anymore I just do a DEL session:sessionid
and all data associated with that hash id is gone. DEL
is atomic and accessing data/writing data to the hash is O(1).
Adding to this answer:
To find first 1000 keys:
EVAL "return redis.call('scan', 0, 'COUNT', 1000, 'MATCH', ARGV[1])" 0 find_me_*
To delete them:
EVAL "return redis.call('del', unpack(redis.call('SCAN', 0, 'COUNT', 1000, 'MATCH', ARGV[1])[2]))" 0 delete_me_*
A version using SCAN rather than KEYS (as recommended for production servers) and --pipe
rather than xargs.
I prefer pipe over xargs because it's more efficient and works when your keys contain quotes or other special characters that your shell with try and interpret. The regex substitution in this example wraps the key in double quotes, and escapes any double quotes inside.
export REDIS_HOST=your.hostname.com
redis-cli -h "$REDIS_HOST" --scan --pattern "YourPattern*" > /tmp/keys
time cat /tmp/keys | perl -pe 's/"/\\"/g;s/^/DEL "/;s/$/"/;' | redis-cli -h "$REDIS_HOST" --pipe
FYI.
redis-cli
keys
(this uses scan
)Maybe you only need to modify capital characters.
scan-match.sh
#!/bin/bash
rcli="/YOUR_PATH/redis-cli"
default_server="YOUR_SERVER"
default_port="YOUR_PORT"
servers=`$rcli -h $default_server -p $default_port cluster nodes | grep master | awk '{print $2}' | sed 's/:.*//'`
if [ x"$1" == "x" ]; then
startswith="DEFAULT_PATTERN"
else
startswith="$1"
fi
MAX_BUFFER_SIZE=1000
for server in $servers; do
cursor=0
while
r=`$rcli -h $server -p $default_port scan $cursor match "$startswith*" count $MAX_BUFFER_SIZE `
cursor=`echo $r | cut -f 1 -d' '`
nf=`echo $r | awk '{print NF}'`
if [ $nf -gt 1 ]; then
for x in `echo $r | cut -f 1 -d' ' --complement`; do
echo $x
done
fi
(( cursor != 0 ))
do
:
done
done
clear-redis-key.sh
#!/bin/bash
STARTSWITH="$1"
RCLI=YOUR_PATH/redis-cli
HOST=YOUR_HOST
PORT=6379
RCMD="$RCLI -h $HOST -p $PORT -c "
./scan-match.sh $STARTSWITH | while read -r KEY ; do
$RCMD del $KEY
done
Run at bash prompt
$ ./clear-redis-key.sh key_head_pattern
I think what might help you is the MULTI/EXEC/DISCARD. While not 100% equivalent of transactions, you should be able to isolate the deletes from other updates.
Please use this command and try :
redis-cli --raw keys "$PATTERN" | xargs redis-cli del
If we want to make sure of atom operation we can try to write a Lua script.
If your Redis version support SCAN
and UNLINK
which is higher than 4.0.0
,I prefer to use SCAN
and UNLINK
instead of Key
and DEL
in the production environment, because Key
and DEL
commands might block
they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.
EVAL "local cursor = 0 repeat local result = redis.call('SCAN', cursor, 'MATCH', ARGV[1]) for _,key in ipairs(result[2]) do redis.call('UNLINK', key) end cursor = tonumber(result[1]) until cursor == 0 " 0 prefix:*
We can change prefix:*
as we want.
nc -v <yourredisserver> 6379
I was able to remove 3M keys out of a 95m key redis instance in 2.5 mins. This is on redis 5 and it would be even faster on 6.
This is not direct answer to the question, but since I got here when searching for my own answers, I'll share this here.
If you have tens or hundreds of millions of keys you have to match, the answers given here will cause Redis to be non responsive for significant amount of time (minutes?), and potentially crash because of memory consumption (be sure, background save will kick in in the middle of your operation).
The following approach is undeniably ugly, but I didn't find a better one. Atomicity is out of question here, in this case main goal is to keep Redis up and responsive 100% of the time. It will work perfectly if you have all your keys in one of databases and you don't need to match any pattern, but cannot use http://redis.io/commands/FLUSHDB because of it's blocking nature.
Idea is simple: write a script that runs in a loop and uses O(1) operation like http://redis.io/commands/SCAN or http://redis.io/commands/RANDOMKEY to get keys, checks if they match the pattern (if you need it) and http://redis.io/commands/DEL them one by one.
If there is a better way to do it, please let me know, I'll update the answer.
Example implementation with randomkey in Ruby, as a rake task, a non blocking substitute of something like redis-cli -n 3 flushdb
:
desc 'Cleanup redis'
task cleanup_redis: :environment do
redis = Redis.new(...) # connection to target database number which needs to be wiped out
counter = 0
while key = redis.randomkey
puts "Deleting #{counter}: #{key}"
redis.del(key)
counter += 1
end
end
I tried most of methods mentioned above but they didn't work for me, after some searches I found these points:
-n [number]
del
but if there are thousands or millions of keys it's better to use unlink
because unlink is non-blocking while del is blocking, for more information visit this page unlink vs delkeys
are like del and is blockingso I used this code to delete keys by pattern:
redis-cli -n 2 --scan --pattern '[your pattern]' | xargs redis-cli -n 2 unlink
If you have spaces in your key names, this will work with MacOS
redis-cli --scan --pattern "myprefix:*" | tr \\n \\0 | xargs -0 redis-cli unlink
Below command worked for me.
redis-cli -h redis_host_url KEYS "*abcd*" | xargs redis-cli -h redis_host_url DEL
This one worked for me but may not be atomic:
redis-cli keys "stats.*" | cut -d ' ' -f2 | xargs -d '\n' redis-cli DEL
I know this one is already answered and works. However, it is not uncommon to work with dockerized Redis. Then the command :
docker exec redis-docker-container-name redis-cli --scan --pattern "*" | while read key; do docker exec redis-docker-container-name redis-cli del "$key"; done
poor man's atomic mass-delete?
maybe you could set them all to EXPIREAT the same second - like a few minutes in the future - and then wait until that time and see them all "self-destruct" at the same time.
but I am not really sure how atomic that would be.
I support all answers related to having some tool or execute Lua expression.
One more option from my side:
In our production and pre-production databases there are thousands of keys. Time to time we need to delete some keys (by some mask), modify by some criteria etc. Of course, there is no way to do it manually from CLI, especially having sharding (512 logical dbs in each physical).
For this purpose I write java client tool that does all this work. In case of keys deletion the utility can be very simple, only one class there:
public class DataCleaner {
public static void main(String args[]) {
String keyPattern = args[0];
String host = args[1];
int port = Integer.valueOf(args[2]);
int dbIndex = Integer.valueOf(args[3]);
Jedis jedis = new Jedis(host, port);
int deletedKeysNumber = 0;
if(dbIndex >= 0){
deletedKeysNumber += deleteDataFromDB(jedis, keyPattern, dbIndex);
} else {
int dbSize = Integer.valueOf(jedis.configGet("databases").get(1));
for(int i = 0; i < dbSize; i++){
deletedKeysNumber += deleteDataFromDB(jedis, keyPattern, i);
}
}
if(deletedKeysNumber == 0) {
System.out.println("There is no keys with key pattern: " + keyPattern + " was found in database with host: " + host);
}
}
private static int deleteDataFromDB(Jedis jedis, String keyPattern, int dbIndex) {
jedis.select(dbIndex);
Set<String> keys = jedis.keys(keyPattern);
for(String key : keys){
jedis.del(key);
System.out.println("The key: " + key + " has been deleted from database index: " + dbIndex);
}
return keys.size();
}
}
Ad of now, you can use a redis client and perform first SCAN (supports pattern matching) and then DEL each key individually.
However, there is an issue on official redis github to create a patter-matching-del here, go show it some love if you find it useful!