49

I am trying to send a simple HTTP POST request, retrieve the response body.Following is my code. I am getting

Error: Incorrect header check

inside the "zlib.gunzip" method. I am new to node.js and I appreciate any help.

;

    fireRequest: function() {

    var rBody = '';
    var resBody = '';
    var contentLength;

    var options = {
        'encoding' : 'utf-8'
    };

    rBody = fSystem.readFileSync('resources/im.json', options);

    console.log('Loaded data from im.json ' + rBody);

    contentLength = Buffer.byteLength(rBody, 'utf-8');

    console.log('Byte length of the request body ' + contentLength);

    var httpOptions = {
        hostname : 'abc.com',
        path : '/path',
        method : 'POST',
        headers : {
            'Authorization' : 'Basic VHJhZasfasNWEWFScsdfsNCdXllcjE6dHJhZGVjYXJk',
            'Content-Type' : 'application/json; charset=UTF=8',
            // 'Accept' : '*/*',
            'Accept-Encoding' : 'gzip,deflate,sdch',
            'Content-Length' : contentLength
        }
    };

    var postRequest = http.request(httpOptions, function(response) {

        var chunks = '';
        console.log('Response received');
        console.log('STATUS: ' + response.statusCode);
        console.log('HEADERS: ' + JSON.stringify(response.headers));
        // response.setEncoding('utf8');
        response.setEncoding(null);
        response.on('data', function(res) {
            chunks += res;
        });

        response.on('end', function() {
            var encoding = response.headers['content-encoding'];
            if (encoding == 'gzip') {

                zlib.gunzip(chunks, function(err, decoded) {

                    if (err)
                        throw err;

                    console.log('Decoded data: ' + decoded);
                });
            }
        });

    });

    postRequest.on('error', function(e) {
        console.log('Error occured' + e);
    });

    postRequest.write(rBody);
    postRequest.end();

}
3
  • Could you post your stack trace? Commented Oct 17, 2013 at 23:15
  • 2
    A little tip: when entering code, use spaces instead of tabs. Makes it much easier for formatting.
    – thtsigma
    Commented Oct 18, 2013 at 6:39
  • I'm using zlib.unzip instead zlib.gunzip
    – Evgenii
    Commented Mar 24, 2017 at 17:55

4 Answers 4

27

response.on('data', ...) can accept a Buffer, not just plain strings. When concatenating you are converting to string incorrectly, and then later can't gunzip. You have 2 options:

1) Collect all the buffers in an array, and in the end event concatentate them using Buffer.concat(). Then call gunzip on the result.

2) Use .pipe() and pipe the response to a gunzip object, piping the output of that to either a file stream or a string/buffer string if you want the result in memory.

Both options (1) and (2) are discussed here: http://nickfishman.com/post/49533681471/nodejs-http-requests-with-gzip-deflate-compression

2

In our case we added 'accept-encoding': 'gzip,deflate' to the headers and code started working (solution credited to Arul Mani):

var httpOptions = {
    hostname : 'abc.com',
    path : '/path',
    method : 'POST',
    headers : {
        ...
        'accept-encoding': 'gzip,deflate'
    }
};
1
  • 1
    I agree. By adding those custom headers in request allow browser to automatically decompress gzipped file on fly without any need for post-fetching decompression using either zlib's gunzip or createGunzip methods
    – Waleed93
    Commented Jun 1, 2020 at 18:53
1

I had this error trying to loop with fs.readdirSync but there is a .Dstore file so the unzip function was applied to it.

Be careful to pass only .zip/gz

import gunzip from 'gunzip-file';

const unzipAll = async () => {
  try {
    const compFiles = fs.readdirSync('tmp')
    await Promise.all(compFiles.map( async file => {
      if(file.endsWith(".gz")){
        gunzip(`tmp/${file}`, `tmp/${file.slice(0, -3)}`)
      }
    }));
  }
  catch(err) {
    console.log(err)
  }
}
0

In addition to @Nitzan Shaked's answer, this might be related to the Node.js version.

What I experienced is that https.request (OP uses http.request, but it may behave the same) already decompresses the data under the hood, so that once you accumulate the chunks into a buffer, all you're left with is to call buffer.toString() (assuming utf8 as an example). I myself experienced it in this other answer and it seems to be related to Node.js version.

I'll end up this answer with a live demo of a similar working code which may come handy for future readers (it queries StackExchange API, gets a gzip compressed chunks, and then decompress it):

  • It includes a code that works on 14.16.0 (current StackBlitz version) - which, as I described, already decompresses the data under the hood - but not on Node.js 15.13.0,
  • It includes a commented-out code that works for Node.js 15.13.0 the latter but not for 14.16.0.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.