15

I am having a really hard time wrapping my head around how to stream data back to my client when using Nodejs/Expressjs.

I am grabbing a lot of data from my database and I am doing it in chunks, I would like to stream that back to the client as I get the data such that I do not have to store the entire dataset in memory as a json object before sending it back.

I would like the data to stream back as a file, ie I want the browser to ask my users what to do with the file on download. I was previously creating a file system write stream and stream the contents of my data to the file system, then when done I would send the file back to the client. I would like to eliminate the middle man (creating tmp file on file system) and just stream data to client.

app.get(
    '/api/export',
    function (req, res, next) {
        var notDone = true;
        while (notDone) {
            var partialData = // grab partial data from database (maybe first 1000 records);

            // stream this partial data as a string to res???

            if (checkIfDone) notDone = false;
        }
    }
);

I can call res.write("some string data"), then call res.end() when I am done. However I am not 100% sure that this is actually streaming the response to the client as I write. Seems like expressjs is storing all the data until I call end and then sending the response. Is that true?

What is the proper way to stream strings chunks of data to a response using expressjs?

0

2 Answers 2

9

The response object is already a writable stream. Express handles sending chunked data automatically, so you won't need to do anything extra but:

response.send(data)

You may also want to check out the built-in pipe method, http://nodejs.org/api/stream.html#stream_event_pipe.

5
  • 7
    response.send(data) is essentially response.end(data), so once you call that, you cannot stream any more data.
    – mscdex
    Commented Aug 8, 2014 at 15:05
  • but I could do res.write(data) a bunch of times right? then res.end() when done. Commented Aug 8, 2014 at 15:56
  • You could, but if you end up writing too much too quickly, node could end up buffering that data into memory until the socket is ready to send some more.
    – mscdex
    Commented Aug 8, 2014 at 16:01
  • 7
    sure but the point is if I write once (all the data), res.end(data) and data is millions of db rows of data I could run out of memory on my server. Better solution seems to be query data in chunks (pages) from the DB and write each small chunk as I get them to the stream. Commented Aug 8, 2014 at 16:07
  • 1
    RE: mscdex's comment - It is recommended to use a proxy like nginx when operating in production. This will provide a termination layer for buffering slow client requests and responses. It allows for configuring limits whereby too-slow clients' connections are dropped.
    – thesmart
    Commented Aug 7, 2017 at 17:14
4

You can do this by setting the appropriate headers and then just writing to the response object. Example:

res.writeHead(200, {
  'Content-Type': 'text/plain',
  'Content-Disposition': contentDisposition('foo.data')
});
var c = 0;
var interval = setInterval(function() {
  res.write(JSON.stringify({ foo: Math.random() * 100, count: ++c }) + '\n');
  if (c === 10) {
    clearInterval(interval);
    res.end();
  }
}, 1000);


// extracted from Express, used by `res.download()`
function contentDisposition(filename) {
  var ret = 'attachment';
  if (filename) {
    filename = basename(filename);
    // if filename contains non-ascii characters, add a utf-8 version ala RFC 5987
    ret = /[^\040-\176]/.test(filename)
      ? 'attachment; filename="' + encodeURI(filename) + '"; filename*=UTF-8\'\'' + encodeURI(filename)
      : 'attachment; filename="' + filename + '"';
  }

  return ret;
}

Also, Express/node does not buffer data written to a socket unless the socket is paused (either explicitly or implicitly due to backpressure). Data buffered by node when in this paused state may or may not be combined with other data chunks that already buffered. You can check the return value of res.write() to determine if you should continue writing to the socket. If it returns false, then listen for the 'drain' event and then continue writing again.

2
  • so res.write() will start sending to client, even before res.end() is called? Hopefully... Commented Aug 8, 2014 at 15:57
  • Yes. It will send it immediately to the socket unless res.write() returns false.
    – mscdex
    Commented Aug 8, 2014 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.