65

I'm writing a crawler which uses Hpricot. It downloads a list of strings from some webpage, then I try to write it to the file. Something is wrong with the encoding:

"\xC3" from ASCII-8BIT to UTF-8

I have items which are rendered on a webpage and printed this way:

Développement

the str.encoding returns UTF-8, so force_encoding('UTF-8') doesn't help. How may I convert this to readable UTF-8?

1
  • 3
    Hpricot is no longer maintained, consider using Nokogiri. Also, you should probably mention what the encoding of the original web page is. Jun 10, 2013 at 12:06

3 Answers 3

72

Your string seems to have been encoded the wrong way round:

"Développement".encode("iso-8859-1").force_encoding("utf-8")
#=> "Développement"
3
  • 1
    It works good for most of cases. But sometimes: U+201C from UTF-8 to ISO-8859-1 in CIDEM / ACC1Ó U+20AC from UTF-8 to ISO-8859-1 in Citi’s Sustainable Development Investments it doesn't. Also some names are converted but wrong and I can't seed it in a database with incomplete multibyte character error message
    – ciembor
    Jun 11, 2013 at 12:36
  • 2
    Sorry, this was not meant as a fix. You should fix the problem by setting/detecting the correct encoding when reading the strings into your app.
    – Stefan
    Jun 11, 2013 at 12:46
  • 2
    There is also the option of using Encoding::UTF_8 instead of using more memory for the "utf-8" string literal (or any other encoding string).
    – Todd
    Feb 17, 2018 at 19:22
61

Seems your string thinks it is UTF-8, but in reality, it is something else, probably ISO-8859-1.

Define (force) the correct encoding first, then convert it to UTF-8.

In your example:

puts "Développement".encode('iso-8859-1').encode('utf-8')

An alternative is:

puts "\xC3".force_encoding('iso-8859-1').encode('utf-8') #-> Ã

If the à makes no sense, then try another encoding.

1
  • Works for pdfs created with Wicked PDF gem Jan 22, 2019 at 17:23
6

"ruby 1.9: invalid byte sequence in UTF-8" described another good approach with less code:

file_contents.encode!('UTF-16', 'UTF-8')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.