99

How can I get the HTML code from a website, save it, and find some text by using a LINQ expression?

I'm using the following code to get the source of a web page:


public static String code(string Url)
{
    HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(Url);
    myRequest.Method = "GET";
    WebResponse myResponse = myRequest.GetResponse();
    StreamReader sr = new StreamReader(myResponse.GetResponseStream(),
        System.Text.Encoding.UTF8);
    string result = sr.ReadToEnd();
    sr.Close();
    myResponse.Close();
    
    return result;
}

How do I find the text within a div in the source of the web page?

5
  • Depends how smart search should be. A simple Contains call might be "good enough."
    – ashes999
    May 20, 2013 at 3:41
  • 5
    Look into using HTMLAgility pack, Fizzler or CSQuery to get the div/text once you have the HTML, anything else is too error prone.
    – jammykam
    May 20, 2013 at 3:43
  • possible duplicate of How can I download HTML source in C# May 20, 2013 at 7:00
  • @GeorgeDuckett That doesn't look like a duplicate of this question, the question you link to is only about retrieving the source, this question is also about querying the DOM. May 20, 2013 at 8:15
  • @Mark: Sorry you're quite right, missed the text at the bottom. May 20, 2013 at 8:17

7 Answers 7

190

Better you can use the Webclient class to simplify your task:

using System.Net;

using (WebClient client = new WebClient())
{
    string htmlCode = client.DownloadString("http://somesite.com/default.html");
}
3
  • Any idea why I get this error? 'System.Net.WebClient': type used in a using statement must be implicitly convertible to 'System.IDisposable' Nov 15, 2013 at 20:31
  • 10
    For the using requirement Clearly shown for everyone to use : +1 Apr 10, 2015 at 19:46
  • For those who are getting http 403 error, add client.Headers.Add("user-agent", "Fiddler"); Replace Fiddler with any text you want. Dec 30, 2020 at 19:26
120

Getting HTML code from a website. You can use code like this:

string urlAddress = "http://google.com";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode == HttpStatusCode.OK)
{
    Stream receiveStream = response.GetResponseStream();
    StreamReader readStream = null;
    if (String.IsNullOrWhiteSpace(response.CharacterSet))
        readStream = new StreamReader(receiveStream);
    else
        readStream = new StreamReader(receiveStream,
            Encoding.GetEncoding(response.CharacterSet));
    string data = readStream.ReadToEnd();
    response.Close();
    readStream.Close();
}

This will give you the returned HTML from the website. But find text via LINQ is not that easy. Perhaps it is better to use regular expression but that does not play well with HTML.

3
  • 4
    The Idea of using regex for html or XML is VERY bad coding practice... Going in Your Way - we should use goto keyword everywhere...
    – Lightning3
    Jan 10, 2015 at 0:24
  • 1
    Actually, using regex to search a precise thing within HTML code can be a very decent solution. Atempting to build a HTML parser/interpretor based on regex, on the other hand, would be pure madness. It all depends on the context and the actual task that needs to be performed, but saying that "regex never plays well with HTML" simply isn't a global, unalinable truth. stackoverflow.com/a/1733489/6838730 Nov 29, 2018 at 16:43
  • @MathieuVIALES programmers can get in a trap where they think something is simple and then it becomes complicated but they have too much investment in their first choice.
    – Sam Hobbs
    Dec 13, 2022 at 6:12
41

Best thing to use is HTMLAgilityPack. You can also look into using Fizzler or CSQuery depending on your needs for selecting the elements from the retrieved page. Using LINQ or Regukar Expressions is just to error prone, especially when the HTML can be malformed, missing closing tags, have nested child elements etc.

You need to stream the page into an HtmlDocument object and then select your required element.

// Call the page and get the generated HTML
var doc = new HtmlAgilityPack.HtmlDocument();
HtmlAgilityPack.HtmlNode.ElementsFlags["br"] = HtmlAgilityPack.HtmlElementFlag.Empty;
doc.OptionWriteEmptyNodes = true;

try
{
    var webRequest = HttpWebRequest.Create(pageUrl);
    Stream stream = webRequest.GetResponse().GetResponseStream();
    doc.Load(stream);
    stream.Close();
}
catch (System.UriFormatException uex)
{
    Log.Fatal("There was an error in the format of the url: " + itemUrl, uex);
    throw;
}
catch (System.Net.WebException wex)
{
    Log.Fatal("There was an error connecting to the url: " + itemUrl, wex);
    throw;
}

//get the div by id and then get the inner text 
string testDivSelector = "//div[@id='test']";
var divString = doc.DocumentNode.SelectSingleNode(testDivSelector).InnerHtml.ToString();

[EDIT] Actually, scrap that. The simplest method is to use FizzlerEx, an updated jQuery/CSS3-selectors implementation of the original Fizzler project.

Code sample directly from their site:

using HtmlAgilityPack;
using Fizzler.Systems.HtmlAgilityPack;

//get the page
var web = new HtmlWeb();
var document = web.Load("http://example.com/page.html");
var page = document.DocumentNode;

//loop through all div tags with item css class
foreach(var item in page.QuerySelectorAll("div.item"))
{
    var title = item.QuerySelector("h3:not(.share)").InnerText;
    var date = DateTime.Parse(item.QuerySelector("span:eq(2)").InnerText);
    var description = item.QuerySelector("span:has(b)").InnerHtml;
}

I don't think it can get any simpler than that.

5
  • What if I want to invoke a specific button on the web page? @jammykam Apr 7, 2017 at 22:26
  • 1
    You can't do that with a screen scraper afaik, you would have to use anything like Selenium to invoke the button.
    – jammykam
    Apr 8, 2017 at 15:20
  • How you install FizzlerEx? I check the link and there is a .zip but don't see any installer Apr 2, 2019 at 18:47
  • FizzlerEx link dead. Also, the github page seems outdated as hell, but is it?
    – IOviSpot
    Dec 4, 2021 at 16:36
  • 1
    @wEight Yes, seems to be dead, stick with [HTML Agility Pack ](html-agility-pack.net)
    – jammykam
    Dec 6, 2021 at 3:03
8

I am using AngleSharp and have been very satisfied with it.

Here is a simple example how to fetch a page:

var config = Configuration.Default.WithDefaultLoader();
var document = await BrowsingContext.New(config).OpenAsync("https://www.google.com");

And now you have a web page in document variable. Then you can easily access it by LINQ or other methods. For example if you want to get a string value from a HTML table:

var someStringValue = document.All.Where(m =>
        m.LocalName == "td" &&
        m.HasAttribute("class") &&
        m.GetAttribute("class").Contains("pid-1-bid")
    ).ElementAt(0).TextContent.ToString();

To use CSS selectors please see AngleSharp examples.

6

Here's an example of using the HttpWebRequest class to fetch a URL

private void buttonl_Click(object sender, EventArgs e) 
{ 
    String url = TextBox_url.Text;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url); 
    HttpWebResponse response = (HttpWebResponse) request.GetResponse(); 
    StreamReader sr = new StreamReader(response.GetResponseStream()); 
    richTextBox1.Text = sr.ReadToEnd(); 
    sr.Close(); 
} 
1
  • 2
    you should add code in your answer instead of an image.
    – A J
    Jun 20, 2016 at 1:33
4

You can use WebClient to download the html for any url. Once you have the html, you can use a third-party library like HtmlAgilityPack to lookup values in the html as in below code -

public static string GetInnerHtmlFromDiv(string url)
    {
        string HTML;
        using (var wc = new WebClient())
        {
            HTML = wc.DownloadString(url);
        }
        var doc = new HtmlAgilityPack.HtmlDocument();
        doc.LoadHtml(HTML);
        
        HtmlNode element = doc.DocumentNode.SelectSingleNode("//div[@id='<div id here>']");
        if (element != null)
        {
            return element.InnerHtml.ToString();
        }   
        return null;            
    }
2

Try this solution. It works fine.

 try{
        String url = textBox1.Text;
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        StreamReader sr = new StreamReader(response.GetResponseStream());
        HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
        doc.Load(sr);
        var aTags = doc.DocumentNode.SelectNodes("//a");
        int counter = 1;
        if (aTags != null)
        {
            foreach (var aTag in aTags)
            {
                richTextBox1.Text +=  aTag.InnerHtml +  "\n" ;
                counter++;
            }
        }
        sr.Close();
        }
        catch (Exception ex)
        {
            MessageBox.Show("Failed to retrieve related keywords." + ex);
        }
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.