4

I am trying to query some URL using WebClient.

I have a collection which I loop through to get the QueryString value, and build final URL, and then pass it on to client.

The very first time it gets executed well, I get proper response, however, when it goes in loop second time, I get the error:

System.Net.WebException --> The remote server returned an error: (403) Forbidden.

If I get response for very first time. Then I should get for rest of the collection too.

Any clue why? What I might be missing?

Below is the code snippet I am using.

 using(System.IO.StreamWriter file = new System.IO.StreamWriter(@"C:\sample.text")) {
    foreach(var f in fileCollections) {
        strFinalURL = string.Empty;

        strFinalURL = "someURL" + f; // f can be considered as querystring param value

        try {
            using(var client = new WebClient()) {

                test = client.DownloadString(strFinalURL);
                if (!test.Contains("somecondition")) {
                    file.WriteLine("");
                }
            }
        } catch (System.Exception ex) {
            Console.WriteLine(ex.Message);
        }
    }
}
6
  • Are all URLs from the same domain? The URL that returns 403, is it accessible via a browser?
    – Tasos K.
    Commented Oct 31, 2013 at 20:25
  • @codingstill yes, all URLs are from same domain and accessible via brower
    – Amit
    Commented Oct 31, 2013 at 20:30
  • 3
    If everything you said is correct, then I suspect server throttling is playing a part. What happens if you slow things down a bit with a strategically placed Thread.Sleep?
    – spender
    Commented Oct 31, 2013 at 20:34
  • Download Fiddler and send an HTTP request using fiddler and look at the data being received and sent.
    – Icemanind
    Commented Oct 31, 2013 at 20:55
  • @spender, Yup Throttling is playing role here.. when I am slowing down the process.. it works... Thanks
    – Amit
    Commented Oct 31, 2013 at 20:56

2 Answers 2

18

Some web servers can block requests based on the user agent string they provide. In your case, you send an empty string as a user agent. Try to add the user agent of a browser, any browser would be fine.

For example:

client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");

For a list of user agents, you can click here.

1
  • 2
    This fixed it for me, would never have thought of that!
    – Simon
    Commented Feb 28, 2016 at 19:04
1

If everything you said is correct, then I suspect server throttling is playing a part. Try adding a Thread.Sleep in your loop to slow things down.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.