0

I have a document that looks similar to this (note the title):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
  <head>
    <title>Sã�ng Title</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  </head>
  <body>
    <div id="container">
      Some Text
    </div>
  </body>
</html>

When i get this document using Nokogiri using this code:

require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open(url).read)

The result from Nokogiri is this:

ruby-1.9.2-p290 :060 > pp doc
#(Document:0x82e5ed2c {
  name = "document",
  children = [
    #(DTD:0x82e5e994 { name = "HTML" }),
    #(Element:0x82e5e0c0 {
      name = "html",
      attributes = [
        #(Attr:0x82e5e05c {
          name = "xmlns",
          value = "http://www.w3.org/1999/xhtml"
          }),
        #(Attr:0x82e5e048 {
          name = "xmlns:fb",
          value = "http://www.facebook.com/2008/fbml"
          })],
      children = [
        #(Element:0x82e5d8dc {
          name = "head",
          children = [
            #(Element:0x82e5d6d4 {
              name = "title",
              children = [ #(Text "Sã")]
              })]
          })]
      })]
  })

To me, it looks like the character AFTER "Sã" causes nokogiri to just choke and think the document has ended. As you can see the #content div is not included at all.

Anyone know how to deal with this situation?

This is killing me... Thank you!!

Edit: Upon further research i've found the actual character causing the choke is a unicode null char "\u0000".

Right now i'm thinking i can do something like this:

page_content = open(url).read
# Remove null character
page_content.gsub!(/\u0000/, '')
Nokogiri::HTML(page_content)

1 Answer 1

1

Are you sure that the character after Sã is a valid UTF-8 character?

Added There are illegal UTF-8 character sequences. To decode UTF-8 manually, try this decoder. You can enter the incoming hex and it will tell you what each individual byte means.

A good overview of UTF-8. UTF-8 code chart

Re: Removing the null character. Your code looks ok, try it out! But in addition, I'd investigate the source of the null in your incoming datastream.

Also, the binary UTF-8 of your Original Post is, in fact, the unknown character symbol--not your original datastream. Here is what is in your post:

53 C3 A3 EF BF BD 6E 67

Here is the decoding:

U+0053 LATIN CAPITAL LETTER S character
U+00E3 LATIN SMALL LETTER A WITH TILDE character (&#x00E3;)
U+FFFD REPLACEMENT CHARACTER character (&#xFFFD;)  # this is the char used when
                                                   # the orig is not understood.
U+006E LATIN SMALL LETTER N character
U+0067 LATIN SMALL LETTER G character
3
  • I am absolutely unsure on that -- is there a way to check? Also, did you mean UTF-8? Sep 27, 2011 at 17:06
  • Updated the initial question with character info. Sep 27, 2011 at 17:44
  • Larry, thanks A TON my friend. These character encoding issues are really tough and sometimes very obscure. I used a combination of what we both said to figure out what the character was and then ultimately remove it. I think it came down to my data source just not being clean and injecting non-utf-8 characters in a doc with a utf-8 charset. Removing those characters cleaned it up. For anyone reading this at a later time the code i posted to clean out the char does in fact work. Sep 29, 2011 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.