0

I'm trying to develop my coding skills, I was trying to design a sort of custom client for viewing google searches.

I don't want to use the google API, mainly because I feel like this will cut out a lot of the interesting programming.

I tried downloading the webpages, with the intent to parse the HTML. However, when I got the search page downloaded, I couldn't find any useful data within like the URLs that I saw on the original page.

Here is an example of a search for the word "file".

http://pastebin.com/xjF2WRRj

Does anyone know a correct way to download the page's html?

Thanks in advance

2 Answers 2

0

What you're trying to do is called web scraping, or trying to pull out content from websites by pretending you loaded a page through a browser, and then accessing the loaded content by looking into and picking out bits and pieces of the page's code. This can work quite well sometimes (there are entire businesses dedicated to web scraping!) and other times not so much - this being one of them.

Since you're essentially ripping the code of the page directly rather than viewing it via a real browser, none of the JavaScript that downloads dynamic content is being executed, and hence, none of the content you're looking for is getting loaded properly. You've downloaded (mostly) the JavaScript from the page itself, which is responsible for loading the content, but it's not getting executed.

Instead, try to download a page with the search results already processed by Google (rather than the results being loaded dynamically), such as this search for "test" on Google. Notice the difference in URLs between that search, and this one, which loads the results for "test" dynamically.

1
  • Thanks a lot, really well explained! Mar 3, 2015 at 23:11
0

Here's how to get the HTML:

Dim WebRequest As HttpWebRequest = TryCast(HttpWebRequest.Create(SearchString), HttpWebRequest)

Dim WR As HttpWebResponse = TryCast(WebRequest.GetResponse(), HttpWebResponse)

Dim receiveStream As Stream = WR.GetResponseStream()

Dim readStream As New StreamReader(receiveStream, Encoding.UTF8)

Dim Page As [String] = readStream.ReadToEnd()

I would use HtmlAgility Pack to parse it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.