234

I'm reading the docs at http://nodejs.org/docs/v0.4.0/api/http.html#http.request, but for some reason, I can't seem to to actually find the body/data attribute on the returned, finished response object.

> var res = http.get({host:'www.somesite.com', path:'/'})

> res.finished
true

> res._hasBody
true

It's finished (http.get does that for you), so it should have some kind of content. But there's no body, no data, and I can't read from it. Where is the body hiding?

1

11 Answers 11

188

http.request docs contains example how to receive body of the response through handling data event:

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/upload',
  method: 'POST'
};

var req = http.request(options, function(res) {
  console.log('STATUS: ' + res.statusCode);
  console.log('HEADERS: ' + JSON.stringify(res.headers));
  res.setEncoding('utf8');
  res.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
  });
});

req.on('error', function(e) {
  console.log('problem with request: ' + e.message);
});

// write data to request body
req.write('data\n');
req.write('data\n');
req.end();

http.get does the same thing as http.request except it calls req.end() automatically.

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/index.html'
};

http.get(options, function(res) {
  console.log("Got response: " + res.statusCode);

  res.on("data", function(chunk) {
    console.log("BODY: " + chunk);
  });
}).on('error', function(e) {
  console.log("Got error: " + e.message);
});
10
  • 19
    For some reason, I had to add res.setEncoding('utf8'); to the http.get example. Otherwise I didn't get HTML in the chunk variable.
    – SSH This
    Nov 28, 2013 at 2:21
  • 1
    @SSHThis that's because they were Buffer objects containing raw data. If you wanted strings from them you could also use chunk.toString(), optionally passing toString and encoding. That said, setEncoding is likely more efficient.
    – skeggse
    Dec 17, 2013 at 19:13
  • 18
    "data" event might be called multiple times and you will get the content piece by piece. Example does not show how to glue them together.
    – Andrej
    Jul 5, 2015 at 0:45
  • 4
    @tfmontague. Agreed! Surprising... so many upvotes for an answer that is flawed at its very foundation.
    – Sunny
    Sep 23, 2015 at 5:20
  • @tfmontague: POST requests typically use a response body, not GET. Post request have a body and GET request does not, but a GET response can have a body.
    – Cyrbil
    Oct 5, 2015 at 10:48
171

I also want to add that the http.ClientResponse returned by http.get() has an end event, so here is another way that I receive the body response:

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/index.html'
};

http.get(options, function(res) {
  var body = '';
  res.on('data', function(chunk) {
    body += chunk;
  });
  res.on('end', function() {
    console.log(body);
  });
}).on('error', function(e) {
  console.log("Got error: " + e.message);
}); 
3
  • 19
    Thanks for that! The 'end' event was crucial for me as I had to process the response body as a whole, rather than in chunks. Jun 25, 2015 at 13:17
  • http.ClientResponse isn't returned by http.get() http.ClientRequest is, according to both the current documentation and the documentation linked to by the original poster.
    – Vince
    Apr 6, 2018 at 9:26
63

Edit: replying to self 6 years later

The await keyword is the best way to get a response from an HTTP request, avoiding callbacks and .then()

You'll also need to use an HTTP client that returns Promises. http.get() still returns a Request object, so that won't work.

  • fetch is a low level client, that is both available from npm and nodeJS 17 onwards.
  • superagent is a mature HTTP clients that features more reasonable defaults including simpler query string encoding, properly using mime types, JSON by default, and other common HTTP client features.
  • axios is also quite popular and has similar advantages to superagent

await will wait until the Promise has a value - in this case, an HTTP response!

const superagent = require('superagent');

(async function(){
  const response = await superagent.get('https://www.google.com')
  console.log(response.text)
})();

Using await, control simply passes onto the next line once the promise returned by superagent.get() has a value.

4
  • 10
    This doesn't answer your original question. In your example code, res is set to the return value of superagent.get(), not http.get(). http.get() returns a http.IncomingMessage that doesn't have a text property. It's not the response object, it's the request object.
    – Vince
    Apr 6, 2018 at 6:34
  • Good point Vince I'll edit the answer to make it cleaner I'm using an HTTP client that supports Promises. Apr 9, 2018 at 10:10
  • Note that superagent is also quite bloated: packagephobia.com/result?p=superagent
    – Jack G
    Jan 27, 2021 at 22:53
  • @JackGiffin Sure, that's because it's old. Newer clients wrap fetch (so they're smaller) and add sensible defaults (so they're more convenient than fetch) Jan 28, 2021 at 11:11
26

The data event is fired multiple times with 'chunks' of the body as they are downloaded and an end event when all chunks have been downloaded.

With Node supporting Promises now, I created a simple wrapper to return the concatenated chunks through a Promise:

const httpGet = url => {
  return new Promise((resolve, reject) => {
    http.get(url, res => {
      res.setEncoding('utf8');
      let body = ''; 
      res.on('data', chunk => body += chunk);
      res.on('end', () => resolve(body));
    }).on('error', reject);
  });
};

You can call it from an async function with:

const body = await httpGet('http://www.somesite.com');
16

The body is not fully stored as part of the response, the reason for this is because the body can contain a very large amount of data, if it was to be stored on the response object, the program's memory would be consumed quite fast.

Instead, Node.js uses a mechanism called Stream. This mechanism allows holding only a small part of the data and allows the programmer to decide if to fully consume it in memory or use each part of the data as its flowing.

There are multiple ways how to fully consume the data into memory, since HTTP Response is a readable stream, all readable methods are available on the res object

  1. listening to the "data" event and saving the chunks passed to the callback

    const chunks = []
    
    res.on("data", (chunk) => {
        chunks.push(chunk)
    });
    
    res.on("end", () => {
        const body = Buffer.concat(chunks);
    });
    

When using this approach you do not interfere with the behavior of the stream and you are only gathering the data as it is available to the application.

  1. using the "readble" event and calling res.read()

    const chunks = [];
    
    res.on("readable", () => {
       let chunk;
       while(null !== (chunk = res.read())){
           chunks.push(chunk)
       }
    });
    
    res.on("end", () => {
        const body = Buffer.concat(chunks);
    });
    

When going with this approach you are fully in charge of the stream flow and until res.read is called no more data will be passed into the stream.

  1. using an async iterator

    const chunks = [];
    
    for await (const chunk of readable) {
        chunks.push(chunk);
    } 
    
    const body = Buffer.concat(chunks);
    

This approach is similar to the "data" event approach. It will just simplify the scoping and allow the entire process to happen in the same scope.

While as described, it is possible to fully consume data from the response it is always important to keep in mind if it is actually necessary to do so. In many cases, it is possible to simply direct the data to its destination without fully saving it into memory.

Node.js read streams, including HTTP response, have a built-in method for doing this, this method is called pipe. The usage is quite simple, readStream.pipe(writeStream);.

for example:

If the final destination of your data is the file system, you can simply open a write stream to the file system and then pipe the data to ts destination.

const { createWriteStream } = require("fs");
const writeStream = createWriteStream("someFile");
res.pipe(writeStream);
3
  • 2
    This well explains why I can't see the response body in the VSC debugger. Thank you! Jun 3, 2021 at 5:20
  • 1
    I think this should be the accepted answer as it succinctly describes how to use the native streaming functionality in Node, rather than requiring an additional dependency. Aug 26, 2021 at 19:02
  • 1
    This has to be the accepted answer! The Buffer.concat(chunks) is the real trick here. Mar 23 at 8:32
12

If you want to use .get you can do it like this

http.get(url, function(res){
    res.setEncoding('utf8');
    res.on('data', function(chunk){
        console.log(chunk);
    });

});
2
  • 2
    The other examples gave me what looked like hex values when I didn't included text with the chunk response. Setting the encoding displayed the JSON document I was looking for. Thank you! Dec 11, 2013 at 6:50
  • @CollinMcGuire that's because they were Buffer objects containing raw data. If you wanted strings from them you could also use chunk.toString(), optionally passing toString and encoding. That said, setEncoding is likely more efficient.
    – skeggse
    Dec 17, 2013 at 19:13
9

You need to add a listener to the request because node.js works asynchronous like that:

request.on('response', function (response) {
  response.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
 });
});
0
4

Just an improved version to nkron responce.

const httpGet = url => {
  return new Promise((resolve, reject) => {
    http.get(url, res => {
      res.setEncoding('utf8');
      const body = [];
      res.on('data', chunk => body.push(chunk));
      res.on('end', () => resolve(body.join('')));
    }).on('error', reject);
  });
};

Appending chunks in an string[] is better for memory usage, the join(''), will allocate new memory only once.

0
2

Needle module is also good, here is an example which uses needle module

var needle = require('needle');

needle.get('http://www.google.com', function(error, response) {
  if (!error && response.statusCode == 200)
    console.log(response.body);
});
1

You can't get the body of the response from the return value of http.get().

http.get() doesn't return a response object. It returns the request object (http.clientRequest). So, there isn't any way to get the body of the response from the return value of http.get().

I know it's an old question, but reading the documentation you linked to shows that this was the case even when you posted it.

0

A portion of Coffee here:

# My little helper
read_buffer = (buffer, callback) ->
  data = ''
  buffer.on 'readable', -> data += buffer.read().toString()
  buffer.on 'end', -> callback data

# So request looks like
http.get 'http://i.want.some/stuff', (res) ->
  read_buffer res, (response) ->
    # Do some things with your response
    # but don't do that exactly :D
    eval(CoffeeScript.compile response, bare: true)

And compiled

var read_buffer;

read_buffer = function(buffer, callback) {
  var data;
  data = '';
  buffer.on('readable', function() {
    return data += buffer.read().toString();
  });
  return buffer.on('end', function() {
    return callback(data);
  });
};

http.get('http://i.want.some/stuff', function(res) {
  return read_buffer(res, function(response) {
    return eval(CoffeeScript.compile(response, {
      bare: true
    }));
  });
});
0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.