0

I would like to web-scrape the html source code of java-script pages that I can´t access without selecting one option in a drop-down list and, after, 'clicking' on links. Spite of not been in java, a simple example can be this:

Web-scrape the main wikipedia pages in all languages available in the drop-down list in the bottom of this url: http://www.wikipedia.org/

To do so, I need to select one language, English for example, and then 'click' in the 'Main Page' link in the left of the new url (http://en.wikipedia.org/wiki/Special:Search?search=&go=Go).

After this step, I would scrape the html source code of the wikipedia main page in English.

Is there any way to do this using R? I have already tried RCurl and XML packages, but it does not work well with the javascript page.

If it is not possible with R, could anyone tell me how to do this with python?

1
  • 1
    I don´t know why someone has downgraded the question without commenting. Actually I am a begginer in R programming and, after celenius answers, I am starting to learn Python. I would be grateful to know if the question is unclear or if it does not show any research effort. Apr 5, 2012 at 2:16

2 Answers 2

3

It's possible to do this using python with the selenium package. There are some useful examples here. I found it helpful to install Firebug so that I could identify elements on the page. There is also a Selenium Firefox plugin with an interactive window that can help too.

import sys
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys   

driver = webdriver.Firefox()
driver.get("http://website.aspx")    

elem = driver.find_element_by_id("ctl00_ctl00")
elem.send_keys( '15' )
elem.send_keys( Keys.RETURN )
1
  • celenius, actually I don´t use python. Do you know any material reference for beginners? Mar 11, 2012 at 22:46
2

Take a look at the RCurl and XML packages for posting form information to the website and then processing the data afterwards. RCurl is pretty cool, but you might have an issue with the HTML parsing because if it isn't standards compliant, the XML package may not want to play nice.

If you are interested in learning Python however, Celenius' example above coupled with beautifulSoup would be what you need.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.