71

I'm using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).

I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it's not exactly what I need.

How can I simply click a button, which then shows the information I need?

Do I have to use an external library like mechanize or lxml?

4 Answers 4

82

Scrapy cannot interpret javascript.

If you absolutely must interact with the javascript on the page, you want to be using Selenium.

If using Scrapy, the solution to the problem depends on what the button is doing.

If it's just showing content that was previously hidden, you can scrape the data without a problem, it doesn't matter that it wouldn't appear in the browser, the HTML is still there.

If it's fetching the content dynamically via AJAX when the button is pressed, the best thing to do is to view the HTTP request that goes out when you press the button using a tool like Firebug. You can then just request the data directly from that URL.

Do I have to use an external library like mechanize or lxml?

If you want to interpret javascript, yes you need to use a different library, although neither of those two fit the bill. Neither of them know anything about javascript. Selenium is the way to go.

If you can give the URL of the page you're working on scraping I can take a look.

4
  • 3
    It's not a real answer to my actual question, but it solves my problem nonetheless. I traced down the site request in Chrome, and found a link which shows the the information in a seperate page. Thanks! But I won't flag your answer as accepted, because others may really need to click a button.
    – naeg
    Commented Jul 13, 2011 at 17:25
  • 2
    @naeg I think the answer summarizes it correctly. You cannot click a button with Scrapy. You can send requests & receive a response. It's upto you to interpret the response with a separate javascript engine.
    – user
    Commented Jul 14, 2011 at 3:50
  • 1
    As it seems Acorn edited his answer, and now it is a full and accepted answer to my question :)
    – naeg
    Commented Jul 14, 2011 at 11:39
  • 1
    You can also use Scrapy-splash : blog.scrapinghub.com/2015/03/02/…
    – Marius
    Commented Apr 2, 2019 at 17:32
23

Selenium browser provide very nice solution. Here is an example (pip install -U selenium):

from selenium import webdriver

class northshoreSpider(Spider):
    name = 'xxx'
    allowed_domains = ['www.example.org']
    start_urls = ['https://www.example.org']

    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self,response):
            self.driver.get('https://www.example.org/abc')

            while True:
                try:
                    next = self.driver.find_element_by_xpath('//*[@id="BTN_NEXT"]')
                    url = 'http://www.example.org/abcd'
                    yield Request(url,callback=self.parse2)
                    next.click()
                except:
                    break

            self.driver.close()

    def parse2(self,response):
        print 'you are here!'
2
  • If you yield the url after clicking the button by selenium to parse2, does it pass on the html with the content that's revealed by the click? Commented Mar 10, 2018 at 14:54
  • @jose I am not sure about that, but it should be easy to test Commented Mar 15, 2018 at 10:07
1

Although it's an old thread I've found quite useful to use Helium (built on top of Selenium) for this purpose and far more easier/simpler than using Selenium. It will be something like the following:

from helium import *

start_firefox('your_url')
s = S('path_to_your_button')
click(s)
...

0

To properly and fully use JavaScript you need a full browser engine and this is possible only with Watir/WatiN/Selenium etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.