17

I just started experimenting with C# WebClient. What I have is the code below which gets html code from a website and writes it in a .txt file. The only problem I have is that some websites require you to accept cookies before you can use the website. What this causes is instead of writing the real website html code to the .txt file, it writes the cookie popup html code.

Code:

string downloadedString;
System.Net.WebClient client;

client = new System.Net.WebClient();
 
//"http://nl.wikipedia.org/wiki/Lijst_van_spelers_van_het_Nederlands_voetbalelftal"
downloadedString = client.DownloadString(textBox1.Text);

using (StreamWriter write = new StreamWriter("Data.txt"))
{
    write.Write(downloadedString);
}

So what is the solution to this? Can somebody direct me to the right path?

2
  • In this particular case there is API mediawiki.org/wiki/API:Main_page that can make automatic download easier.
    – Andrey
    Jan 27, 2013 at 19:38
  • You must mean the .NET WebClient class, since there's no "C# WebClient". Jan 27, 2013 at 19:52

3 Answers 3

29

Usage :

        CookieContainer cookieJar = new CookieContainer();
        cookieJar.Add(new Cookie("my_cookie", "cookie_value", "/", "mysite"));

        CookieAwareWebClient client = new CookieAwareWebClient(cookieJar);

        string response = client.DownloadString("http://example.com/response_with_cookie_only.php");

public class CookieAwareWebClient : WebClient
{
    public CookieContainer CookieContainer { get; set; }
    public Uri Uri { get; set; }

    public CookieAwareWebClient()
        : this(new CookieContainer())
    {
    }

    public CookieAwareWebClient(CookieContainer cookies)
    {
        this.CookieContainer = cookies;
    }

    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest request = base.GetWebRequest(address);
        if (request is HttpWebRequest)
        {
            (request as HttpWebRequest).CookieContainer = this.CookieContainer;
        }
        HttpWebRequest httpRequest = (HttpWebRequest)request;
        httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        return httpRequest;
    }

    protected override WebResponse GetWebResponse(WebRequest request)
    {
        WebResponse response = base.GetWebResponse(request);
        String setCookieHeader = response.Headers[HttpResponseHeader.SetCookie];

        //do something if needed to parse out the cookie.
        if (setCookieHeader != null)
        {
            Cookie cookie = new Cookie(); //create cookie
            this.CookieContainer.SetCookies(request.RequestUri, setCookieHeader);
        }
        
        return response;
    }
}

You will see two overridden methods for GetWebRequest and GetWebResponse. These methods can be overridden to handle the cookie container.

4
  • Is there a reason why you check setCookieHeader != null twice?
    – easuter
    Apr 28, 2015 at 22:33
  • 3
    Exception::: parameter cant be empty . paramenter name: cookie.domain
    – bh_earth0
    Jan 6, 2017 at 14:53
  • Why the public Uri property? Aug 4, 2017 at 9:23
  • replace this.CookieContainer.Add(cookie); with this.CookieContainer.SetCookies(request.RequestUri, setCookieHeader);
    – Hichem
    Jun 29, 2021 at 10:18
5

Just store cookie string from headers into your local session _cookies string

if (System.Web.HttpContext.Current.Session["cookie"] != null)
            _cookies = System.Web.HttpContext.Current.Session["cookie"].ToString(); 

     using (WebClient wc =  new WebClient())
        {

            wc.Headers.Add("Cookie", _cookies);
             string HtmlResult = wc.UploadString(bridge_url, myParameters);
            _cookies = wc.ResponseHeaders["Set-Cookie"];
            Debug.WriteLine("Headers" + _cookies); 

            System.Web.HttpContext.Current.Session["cookie"] = _cookies;

        } 
3

This may be a close duplicate of How can I get the WebClient to use Cookies?

The question I referenced above is for VB.NET, but the mechanism should be the same for C#. I suspect the behavior you are seeing is the web-site is sending a cookie and then requesting it back, but your client is not setup to return the cookie to the server, so it interprets that as you 'not accepting cookies.'

Have you used a analysis tool like Fiddler to analyze what is being communicated to/from your client?

You may also have to send a particular HTTP header to indicate you accept cookies, but I don't recall that being required in my past experience.

2
  • what u mean by Original question is for VB.NET,? Jan 27, 2013 at 19:49
  • 1
    The existing question on Stackoverflow that I linked in my answer was for VB.BET. Since you are using C#, the syntax in the answer to the previous question would differ, but the methodology would be the same since both are using .NET classes. I edited my answer to clarify the point.
    – maelstrom
    Jan 27, 2013 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.