0

I am making a desktop yellowpage application. I can access all countries yellowpage site but not australian site. I dont know why?

Here is the code

class Program
{
    static void Main(string[] args)
    {
        WebClient wb = new WebClient();
        wb.Headers.Add("user-agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)");
        string html = wb.DownloadString("http://www.yellowpages.com.au");
        Console.WriteLine(html);
    }
}

For all other site I get html of the website for australian site I get null. i even tried httpwebrequest also.

Here is the yellowpage australian site: http://www.yellowpages.com.au

Thanks in advance

5
  • No idea why. What errors/exceptions are you getting?
    – Oded
    Jun 4, 2011 at 18:13
  • I didnt goy any errors or exceptions
    – Ajit
    Jun 4, 2011 at 18:15
  • If you post code, post your real code, not something that won't even compile.
    – svick
    Jun 4, 2011 at 18:17
  • I think above code can be compiled.
    – Ajit
    Jun 4, 2011 at 18:22
  • And now that it works, please note the copyright notice near the end of that html string.
    – H H
    Jun 4, 2011 at 18:54

3 Answers 3

3

It looks like that website will only send over gzip'ed data. Try switching to HttpWebRequest and using auto decompression:

var request = (HttpWebRequest)WebRequest.Create("http://www.yellowpages.com.au");
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)";
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
0
1

In addition to @bkaid's correct (and upvoted) answer, you can use your own class inherited from WebClient to uncompress/handle gzip compressed html:

public class GZipWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
        request.AutomaticDecompression = DecompressionMethods.GZip | 
                                         DecompressionMethods.Deflate;
        return request;
    }
}

Having done this, the following works just fine:

WebClient wb = new GZipWebClient();
string html = wb.DownloadString("http://www.yellowpages.com.au");
1
  • Interesting, I didn't know you could do that.
    – svick
    Jun 5, 2011 at 3:33
0

When I view the transfer from that website in Wireshark, it says it's a malformed HTTP packet. It says it uses chunked transfer, then says the following chunk has 0 bytes and then sends the code of the website. That's why WebClient returns an empty string (not null). And I think it's correct behavior.

It seems browsers ignore this error and so they can display the page properly.

EDIT:

As bkaid pointed out, the server seems to handle send correct gziped response. The following code works for me:

WebClient wb = new WebClient();
wb.Headers.Add("Accept-Encoding", "gzip");
string html;
using (var webStream = wb.OpenRead("http://www.yellowpages.com.au"))
using (var gzipStream = new GZipStream(webStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(gzipStream))
    html = streamReader.ReadToEnd();
1
  • yeah.. I saw the chunked transfer and it says content length 0 when I viewed the web response through debugging mode in visual studio. It says content length -1 and it doesnt show content encoding .. Hence I didnt understand what is happening there.. Anyway thank you it worked.
    – Ajit
    Jun 5, 2011 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.