38

I'm using redis in my python application to store simple values like counters and time stamp lists, but trying to get a counter and comparing it with a number I came across a problem.

If I do:

import redis
...
myserver = redis.Redis("localhost")
myserver.set('counter', 5)

and then try to get that value like this:

if myserver.get('counter') < 10:
     myserver.incr('counter')

then I get a type error in the if statement because I'm comparing '5' < 10, which means I'm storing an integer value and getting a string one (which can be considered as a different value).

My question is: is this supposed to work like that? I mean its a very basic type, I understand if I have to parse objects but an int? Seems that I'm doing something wrong.

Is there any configuration I'm missing?

Is there any way to make redis return the right type and not always a string? I say this because its the same for lists and datetimes or even floating point values.

Could this be a problem with the redis-py client I'm using and not redis itself?

2
  • I've been messing around with the src/redis-cli that comes with their distribution. Unfortunately it looks like redis stores integers as strings, and infers there types when doing incr/decr operations. Oct 25, 2012 at 2:54
  • 1
    you can use if int(myserver.get('counter'))<10.
    – Mayank
    Jul 14, 2018 at 17:50

8 Answers 8

19

Technically speaking you need to take care of that on your own.

However, have a look at this link, especially at the part of their README that refers to parsers and response callbacks, maybe that's something you can use. Question would be whether this is an overkill for you or not.

1
  • 1
    thank you i checked the parsers but they seem to overcomplicate stuff for what i want to do, in the end i have decided to use cpickle to handle the serialization of data
    – jeruki
    Oct 25, 2012 at 16:40
17

As @favoretti said, response callbacks will do the trick. It's not complicate at all, just one line and all will be taken care of.

In [2]: import redis
In [3]: r = redis.Redis()
In [10]: r.set_response_callback('HGET', float)
In [11]: r.hget('myhash', 'field0')
Out[11]: 4.6

for hmget, it returns a list of strings, not one single string, so you need to construct a little more comprehensive callback function:

In [12]: r.set_response_callback('HMGET', lambda l: [float(i) for i in l])

In [13]: r.hmget('myhash', 'field0')
Out[13]: [4.6]

same for hgetall.

1
  • 2
    Tried to set float callback for "HMGET" and "HGETALL" it fails unfortunately. I guess it tries to transform the hash key names.
    – monkut
    Dec 19, 2013 at 6:44
9

Though leveraging set_response_callback is fine for simple data types, if you're wondering about the fastest-and-easiest route to storing things like dictionaries, list, tuples – and preserving the python-native datatypes they may-or-may-not contain – I recommend using python's built-in pickle library:

# Imports and simplified client setup
>>> import pickle
>>> import redis
>>> client = redis.Redis()
# Store a dictionary
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> client.set('TestKey', pickle.dumps(to_store))
True
# Retrieve the dictionary you just stored.
>>> retrieved = pickle.loads(client.get('TestKey'))
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}

Here's a simple client that will reduce the pickle boilerplate in the above example and provide you with a clean interface for storing and retrieving native python datatypes to and from Redis:

"""Redis cache."""
import pickle
import redis

redis_host = redis.Redis()


class PythonNativeRedisClient(object):
    """A simple redis client for storing and retrieving native python datatypes."""

    def __init__(self, redis_host=redis_host):
        """Initialize client."""
        self.client = redis_host

    def set(self, key, value, **kwargs):
        """Store a value in Redis."""
        return self.client.set(key, pickle.dumps(value), **kwargs)

    def get(self, key):
        """Retrieve a value from Redis."""
        val = self.client.get(key)
        if val:
            return pickle.loads(val)
        return None

redis_client = PythonNativeRedisClient()

Usage:

>>> from some_module import redis_client
>>> to_store = {'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
>>> redis_client.set('TestKey', to_store)
True
>>> retrieve = redis_client.get('TestKey')
{'a': 1, 'b': 'A string!', 'c': [1, True, False, 14.4]}
6

Looks like that is how redis stores its data:

redis 127.0.0.1:6379> set counter 5
OK
redis 127.0.0.1:6379> type counter
string
redis 127.0.0.1:6379> incr counter
(integer) 6
redis 127.0.0.1:6379> type counter
string

If you really want to, you could probably monkeypatch the redis-py client to infer data types.

1
  • apparently it is i just thought that redis-py would have a mecanism to work with that, thanks
    – jeruki
    Oct 25, 2012 at 15:39
5

You can set decode_respone like True

redis.StrictRedis(host="localhost", port=6379, db=0, decode_responses=True)
2
  • 3
    it seems not working >>> r= redis.StrictRedis(decode_responses=True) >>> r.set('world', 123) True >>> r.get('world') '123'
    – Tsonglew
    Nov 13, 2019 at 14:05
  • 1
    This only converts return value from bytes to str. If you store a float or integer, you will still get str response!
    – Sarang
    Apr 12, 2022 at 10:34
4

Depending on needs, we have 3 ways to store data into Redis

String

  • All data are stored and read back as bytes.
  • This is the native way method that used by Redis.
  • We could use redis-cli to inspect and manipulate the data
  • However, data may not be stored efficiently. For example 12345 is stored as string using at least 5 bytes even it could be represented by a 2-byte unsigned short.
  • You could limited to use various data structures offered by Redis.
  • The data stored within Redis is portable across different versions of Python and programming languages.

Pickle

  • All Python data are converted into binary data using pickle library before stored into Redis.
  • Data must be converted back to Python data types using pickle library before could be used.
  • It could model complex and dynamic data structures not offered by Redis.
  • Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
  • It uses more memory within Redis as pickle includes meta data, references in its output.
  • It is not possible to use redis-cli to manipulate the data, such as INCR.
  • The data stored within Redis is not portable across different versions of Python and programming languages.

Struct

  • All Python data are converted into binary data using struct library before stored into Redis.
  • Data must be converted back to Python data types using struct library before could be used.
  • It could model complex and static data structures not offered by Redis.
  • Once the data is converted back to Python data types, We could use powerful Python to manipulate the data.
  • It uses the least memory out of these 3 methods
  • It is not possible to use redis-cli to manipulate the data, such as INCR.
  • The data stored within Redis is portable across different versions of Python and programming languages.
from redis import Redis
import sys
import pickle
import struct

redis = Redis()

# Data
d = [1234, 3.1415, b'Good', True]
k = ['int', 'float', 'byte', 'bool']

## Using String
print(f'String\n{"_"*50}')
# Need to convert True to string or int or float
d2 = [1234, 3.1415, b'Good', int(True)]
redis.hset('k0', mapping=dict(zip(k,d2)))
print(redis.hmget('k0', *k))
print()


## Using Pickle
print(f'Pickle\n{"_"*50}')
packed = pickle.dumps(d)
print(f'Pickled data occupies {sys.getsizeof(packed)} bytes')
print(packed)
redis.set('k1', packed)
print(f'{pickle.loads(redis.get("k1"))}')
print()


## Using Struct
print(f'Struct\n{"_"*50}')
s = struct.Struct('H f 10s ?')
print(f'The structure occupies {s.size} bytes.')
packed = s.pack(*d)
print(f'{packed}')
redis.set('k2', s.pack(*d))
print(f'{s.unpack(redis.get("k2"))}')

Expected output:

String
__________________________________________________
[b'1234', b'3.1415', b'Good', b'1']

Pickle
__________________________________________________
Pickled data occupies 69 bytes
b'\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(M\xd2\x04G@\t!\xca\xc0\x83\x12oC\x04Good\x94\x88e.'
[1234, 3.1415, b'Good', True]

Struct
__________________________________________________
The structure occupies 19 bytes.
b'\xd2\x04\x00\x00V\x0eI@Good\x00\x00\x00\x00\x00\x00\x01'
(1234, 3.1414999961853027, b'Good\x00\x00\x00\x00\x00\x00', True)
1

Here is my test. Two redis connections: one returns int type, the other float

import redis

age_field = redis.Redis()
age_field.set_response_callback('HGET', int)
age_field.hget('foo', 'age')
# OUT: 40
a =age_field.hget('foo', 'age')
type(a)
# OUT: <type 'int'>

gpa_field = redis.Redis()
gpa_field.set_response_callback('HGET', float)
gpa_field.hget('foo', 'gpa')
# OUT: 2.5
b = gpa_field.hget('foo', 'gpa')
type(b)
# OUT: <type 'float'>
0

one solution could be to use a json.loads() to return int or float as a correct type, instead of str.

import json
import redis

redis = redis.from_url("redis://************")

def get(key, decode=True):
    """set decode to False when value stored as a string"""
    value = redis.get(key)
    if not decode:
        return value
    if value is not None:
        try:
            return json.loads(value)
        except json.decoder.JSONDecodeError:
            # not containing a JSON document
            return value

redis.set("foo", 2)
redis.set("bar", 2.75)

get("foo")
# 2
get("bar")
# 2.75

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.