I am having a really hard time wrapping my head around how to stream data back to my client when using Nodejs/Expressjs.
I am grabbing a lot of data from my database and I am doing it in chunks, I would like to stream that back to the client as I get the data such that I do not have to store the entire dataset in memory as a json object before sending it back.
I would like the data to stream back as a file, ie I want the browser to ask my users what to do with the file on download. I was previously creating a file system write stream and stream the contents of my data to the file system, then when done I would send the file back to the client. I would like to eliminate the middle man (creating tmp file on file system) and just stream data to client.
app.get(
'/api/export',
function (req, res, next) {
var notDone = true;
while (notDone) {
var partialData = // grab partial data from database (maybe first 1000 records);
// stream this partial data as a string to res???
if (checkIfDone) notDone = false;
}
}
);
I can call res.write("some string data"), then call res.end() when I am done. However I am not 100% sure that this is actually streaming the response to the client as I write. Seems like expressjs is storing all the data until I call end and then sending the response. Is that true?
What is the proper way to stream strings chunks of data to a response using expressjs?