9

I'm doing the following request (using request/request) against a web service:

return request.postAsync({
    url,
    charset: 'Cp1252', // I also tried utf-8
    encoding: null, //
    // I also tried Cp1252 -> unknown encoding,
    // I also tried utf-8 and nothing at all
    headers: {
         "Accept": "application/octet-stream, text, text/plain, text/xml",
         "Accept-Encoding": "UTF-8",
         'Content-Type': "text/plain; charset=Cp1252;", // also tried utf-8, ISO-8859-1
         "User-Agent": "me"
    }
}).spread((res, body) => {
    body = body.toString();  // I also tried without toString();
    let ws = fs.createWriteStream('hhh.csv');
    ws.write(body);
    ws.end();

Whatever I do, umlauts are turned into .

Those are the headers the web service sends back:

'content-type': 'text; charset=Cp1252',
'content-length': '1895980',
vary: 'Accept-Encoding,User-Agent'

I'm trying this for days with no luck at all. What am I doing wrong?

Here's a list of question/answers that didn't solve my problem so far:

Can it be that one of the following causes my input string to not be UTF-8?

let hash = crypto.createHmac("sha256", this.options.Signer);
this.query['Signature'] = hash.update(stringToSign).digest("base64");

signer is a string containing 0-9, a-z, A-Z, +, and /.

this.query['Signature'] is part of the URL.

13
  • What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? – zerkms Oct 5 '15 at 21:53
  • In the place where the umlaut is "still" umlaut. – zerkms Oct 5 '15 at 21:56
  • "node js request turns umlauts to �" "Whatever I do, umlauts are turned into �." --- what these phrases mean then? – zerkms Oct 5 '15 at 21:58
  • What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? – zerkms Oct 5 '15 at 22:01
  • 1
    Have you tried changing the defaultEncoding of your writeStream? i.e. fs.createWriteStream('hhh.csv', {defaultEncoding: 'utf8'}); – Yan Foto Oct 5 '15 at 22:03
7

I finally solved it, using iconv-lite and setting request's encoding¹ to null, making it return the body as a Buffer. Here is my now working configuration:

return request.getAsync({
        url,
        encoding: null,
        headers: {
            "Accept": "text, text/plain, text/xml",
            "Accept-Encoding": "UTF-8",
            'Content-Type': "text/plain; charset=utf-8;",
            "User-Agent": "me"
        }
    }).spread((res, body) => {
        let a = iconv.decode(new Buffer(body), 'cp1252');
        // now a is holding a string with correct Umlauts and ß
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.