24

I am trying to save a binary object in redis and then serve it back as an image.

Here is the code I am using to save the data:

var buff=new Buffer(data.data,'base64');
client.set(key,new Buffer(data.data,'base64'));

Here is the code to dump the data out:

client.get(key,function(err,reply){
        var data = reply;
        response.writeHead(200, {"Content-Type": "image/png"});
        response.end(data,'binary');

});

The first few byte of the data seem to be corrupted. The magic number is incorrect.

Did some experimenting:

when I do the following:

var buff=new Buffer(data.data,'base64');
console.log(buff.toString('binary'));

I get this:

0000000: c289 504e 470d 0a1a 0a00 0000 0d49 4844

when I do this

 var buff=new Buffer(data.data,'base64');
 console.log(buff);

I get the following:

Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00

I am not sure where the c2 is coming from

4
  • haven't worked with images this way, but isn't supposed to "decode" what was encoded in base64?
    – Gntem
    Dec 22, 2013 at 21:34
  • What do you get if you console.log(reply)? Can you also elaborate on what you are trying accomplish by putting the images in Redis? It is not the best way to store large amounts of binary data. Dec 22, 2013 at 23:25
  • data comes in as base64 trying to save the binary file Dec 23, 2013 at 2:56
  • Follow up to this question: I'm trying to do basically the same thing, and when I read it out with get, the string length is different even when I'm writing the same thing. Apr 2, 2015 at 13:52

7 Answers 7

28

The problem is that the Redis client for Node converts responses to JavaScript strings by default.

I solved this by setting the return_buffers option to true when creating the client.

var client = redis.createClient(7000, '127.0.0.1', {'return_buffers': true});

See here for more details.

3

The problem with return_buffers is when you are not using pure buffer data's then you'll have to do something to convert other buffers to strings. While detect_buffers could be an option it is too unreliable.

If you don't mind the extra computing cycles. You could also try:

// convert your buffer to hex string
client.set(key, mybuffer.toString('hex'));
// create a new buffer from hex string
client.get(key, function (err, val) {
   var bin = new Buffer(val, 'hex');
});
1
  • 1
    In most cases, base64 is a better option since it will produce a smaller message than hex (at a cost of more expensive encoding).
    – Gajus
    Apr 13, 2017 at 15:00
3

I ran in to a similar issue and in my case the corruption was with how I was getting data from Redis into Nodejs. In my case the value from Nodejs was stored correctly but when it came into node it was being converted to a ASCII string. I was able to force it to a buffer with the following line.

let file = await r.get(redis.commandOptions({ returnBuffers: true }), `zip/${req.params.taskId}`)
2

I was unable to figure out how to get the binary string to store.

Here is my workaround:

Where data is the data in base64 string

client.set(count,data);

to serve the data:

 client.get(last,function(err,reply){
   var data = reply;
   response.writeHead(200, {"Content-Type": "image/png"});
   var buff=new Buffer(data,'base64');
   response.end(buff);
});

This isn't ideal since you have to do the conversion every time, but it seem to work.

1
  • Please note, that new Buffer() is deprecated. Use: Buffer.from('some-data', 'base64'); Oct 1, 2020 at 22:14
1

This post is old but just an updated answer. As another answer stated all Node.js Redis libraries return data as utf8 strings because thats what people generally need. Node.js Redis libraries like ioRedis also provide alternatives to fetch data as a buffer. For example redis.getBuffer instead of redis.get.

0

What worked for me is to use data.toString('binary') if it is a Buffer. Also make sure not to reinterpret it as utf-8, but also as binary.

0

I found this article on the topic which explains the implications of doing this, which I highly recommend reading before going down this path (regardless of programming language).

http://qnimate.com/storing-binary-data-in-redis/

In summary, since Redis is an in-memory key/value store, you would be wise to not store potentially large values such as images inside Redis as you will quickly use up available memory and degrade the performance of your Redis instance. It is better to store the location of the file in Redis rather than the file itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.