15

Trying to use linux curl to download an xml file from an url.

Pretty sure that the xml is encoded in UTF-8,

suspecting curl -o doesnt save as UTF-8.

Is there anyway to force save to UTF-8 with curl ?


Thanks for the suggestion, what i found out:

Because the xml feed is dynamic, not all the time it contain any utf-8 characters. Sometimes it doesnt have utf-8 character in the whole content at all even though it is set as utf-8 in the xml encoding and header content type: charset=utf-8. When it contain a utf-8 character at least, it will be save as utf-8.

When this happen, curl doesn't download as utf-8, which makes sense as there are no utf-8 chars, why is there a need to store as utf-8.

This is damn tricky, some validator has to valid against utf-8 hence i still need a solution to force it to utf8 because by default all my xml shld be in utf8-encoding.

tried the suggested by using iconv f iso8859-1 utf-8 doesnt work for this case as i am suspecting it is not in iso8859-1 either.

Still need a better solution.

3
  • I don't believe curl makes any modifications at all to what it fetches, and it certainly doesn't re-encode text.
    – Wooble
    Apr 16, 2012 at 10:23
  • hi thanks... however i'm not looking re-encode data, just wondering why it is not saving as utf-8 while the contents are alr utf-8
    – flyclassic
    Apr 17, 2012 at 1:58
  • What makes you think it's not saving as utf-8? What are you actually trying to accomplish?
    – Wooble
    Apr 17, 2012 at 10:32

2 Answers 2

17

curl does not do any conversion of the file it downloads. If the HTTP server serves you the XML in another encoding (e.g., ISO8859-1) that his how curl will save it to disk too.

To workaround your problem, you can use "iconv" as follows:

curl URL | iconv -f iso8859-1 -t utf-8 > output.xml

Hope this help.

4
  • i've tried this way, but it is still downloaded not downloaded as utf-8. i'm pretty sure the xml contents are in utf-8 as i've developed it and checked with mb_check_encoding it returns true with utf-8. How can i check the file encoding standard in linux? i'm not quite sure if it turns out to be iso8859-1 that's why iconv doesnt work? file -bi doesnt work for me..
    – flyclassic
    Apr 17, 2012 at 2:40
  • You can check the encoding of a file by typing file FILENAME. Please post the output of file (against the XML downloaded with curl, without iconv) here. Apr 17, 2012 at 8:08
  • If you're on a mac, don't want to rely on iconv, and don't want to hardcode assumptions about the incoming text encoding, you can get the same effect with curl URL | textutil -cat txt -stdin -stdout -encoding utf-8
    – algal
    Mar 13, 2014 at 21:56
  • @algal your method fixed utf8 issue for downloading html, but problem is it returns text only w/o html. Not sure about xml files. May 14, 2017 at 19:13
7

Have you tried adding the Accept-Charset header? I had a similar issue downloading a file which was downloading with the wrong encoding. When I set the Accept-Charset header it works:

curl -H "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" URL | iconv -f iso8859-1 -t utf-8 > output.xml

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.