22

I want to extract a couple of links from an html page downloaded from the internet, I think that using linq to XML would be a good solution for my case.
My problem is that I can't create an XmlDocument from the HTML, using Load(string url) didn't work so I downloaded the html to a string using:

public static string readHTML(string url)
    {
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
        HttpWebResponse res = (HttpWebResponse)req.GetResponse();
        StreamReader sr = new StreamReader(res.GetResponseStream());

        string html = sr.ReadToEnd();
        sr.Close();
        return html;
    }

When I try to load that string using LoadXml(string xml) I get the exception

'--' is an unexpected token. The expected token is '>'

What way should I take to read the html file to a parsable XML

2
  • 6
    HTML need not necessarily be a valid XML. HTML is based on SGML, which itself is a superset (kind of) of XML. Hence you need a special HTML parser, not a generic XML parser. Commented Mar 29, 2011 at 12:06
  • More of the same of the answers below and comment above. HTML is not XML Commented Mar 29, 2011 at 12:09

5 Answers 5

16

HTML simply isn’t the same as XML (unless the HTML actually happens to be conforming XHTML or HTML5 in XML mode). The best way is to use a HTML parser to read the HTML. Afterwards you may transform it to Linq to XML – or process it directly.

13

I haven't used it myself, but I suggest you take a look at SgmlReader. Here's a sample from their home page:

// setup SgmlReader
Sgml.SgmlReader sgmlReader = new Sgml.SgmlReader()
{
    DocType = "HTML",
    WhitespaceHandling = WhitespaceHandling.All,
    CaseFolding = Sgml.CaseFolding.ToLower,
    InputStream = reader
};

// create document
XmlDocument doc = new XmlDocument()
{
    PreserveWhitespace = true,
    XmlResolver = null
};
doc.Load(sgmlReader);
return doc;
1
  • +1 I have used SGMLReader for many years (since it was introduced). It is very robust and can handle some very rotten malformed HTML. Commented Nov 14, 2016 at 21:17
8

If you want to extract some links from a page, as you mentioned, try using HTML Agility Pack.

This code gets a page from the web and extracts all links:

HtmlWeb web = new HtmlWeb();  
HtmlDocument document = web.Load("http://www.stackoverflow.com");  
HtmlNode[] links = document.DocumentNode.SelectNodes("//a").ToArray(); 

Open an html file from disk and get URL for specific link:

HtmlDocument document2 = new HtmlDocument();  
document2.Load(@"C:\Temp\page.html")  
HtmlNode link = document2.DocumentNode.SelectSingleNode("//a[@id='myLink']");
Console.WriteLine(link.Attributes["href"].Value);
3

HTML is not XML. HTML is based on SGML, and as such does not ensure that the markup is well-formed XML (XML is a subset of SGML itself). You can only parse XHTML, i.e. XML compatible HTML, as XML. But of course that is not the case for most of the websites.

To work with HTML, you need to use a HTML parser.

1
  • If I could mark two answers I would mark both of yours answers.
    – Ziv
    Commented Mar 29, 2011 at 17:44
-2

If you know the nodes you're interested in I would use regex to extract the links from the string.

2
  • 8
    Regular Expressions are almost never a sensible approach to parsing HTML or XML
    – Nic Gibson
    Commented Mar 29, 2011 at 13:55
  • 1
    I have thought of that but I never learned regex and it's too big of a subject to learn for such a small task.
    – Ziv
    Commented Mar 29, 2011 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.