2

The following Scrapy CrawlSpider class code is for scraping links via following pagination from the data.ok.gov page.

class OklahomaFinanceSpider(CrawlSpider):
    name = "OklahomaFinanceSpider"
    allowed_domains = ["data.ok.gov"]
    start_urls = [
        "http://data.ok.gov/browse?f[0]=bundle_name%3ADataset&f[1]=im_field_categories%3A4191"
        ] 

    rules = (
    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//li[@class="pager-next"]',)), callback="parse_page", follow= True),
) 
def parse_page(self, response): 

        for href in response.xpath('//*[contains(concat(" ", normalize-space(@class), " "),"search-results apachesolr_search-results")]/h3/a/@href'):
        url = response.urljoin(href.extract())
        yield scrapy.Request(url, callback=self.parse_dir_contents)   

However, the first page is not being scraped. What mistake am I making with the Rules?

3
  • 1
    If you're talking about parsing the pages fetched from the start_urls, check parse_start_url. You can set parse_start_url = parse_page and you should be good Nov 20, 2015 at 19:25
  • I did add parse_start_url = parse_page after defining the rules, but in terminal, the error was "NameError: name 'parse_page' is not defined" . Any idea why I am getting it? Nov 20, 2015 at 19:34
  • 1
    you can set parse_start_url = parse_page after the def parse_page(self, response): block Nov 20, 2015 at 19:36

1 Answer 1

0

Posting @paul trmbrth comment as an answer here.

For parsing the pages fetched at the start_urls, set parse_start_url = parse_page after after the parse_page(self, response) definition.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.