2

I am trying to crawl Booking.Com. The spider opens and closes without opening and crawling the url.[Output][1] [1]: https://i.stack.imgur.com/9hDt6.png I am new to python and Scrapy. Here is the code I have written so far. Please point out what I am doing wrong.

import scrapy
import urllib
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import Selector
from scrapy.item import Item
from scrapy.loader import ItemLoader
from CinemaScraper.items import CinemascraperItem


class trip(CrawlSpider):
 name="tripadvisor"

def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)


def parse(self, response):
        reviewsurl = response.xpath('//a[@class="show_all_reviews_btn"]/@href')
        url = response.urljoin(reviewsurl[0].extract())
        self.pageNumber = 1
        return scrapy.Request(url, callback=self.parse_reviews)


def parse_reviews(self, response):
     for rev in response.xpath('//li[starts-with(@class,"review_item")]'):
            item =CinemascraperItem()
            #sometimes the title is empty because of some reason, not sure when it happens but this works
            title = rev.xpath('.//*[@class="review_item_header_content"]/span[@itemprop="name"]/text()')
            if title:
                item['title'] = title[0].extract()
                positive_content = rev.xpath('.//p[@class="review_pos"]//span/text()')
                if positive_content:
                    item['positive_content'] = positive_content[0].extract()
                negative_content = rev.xpath('.//p[@class="review_neg"]/span/text()')
                if negative_content:
                    item['negative_content'] = negative_content[0].extract()
                item['score'] = rev.xpath('./*[@class="review_item_header_score_container"]/span')[0].extract()
                #tags are separated by ;
                item['tags'] = ";".join(rev.xpath('.//ul[@class="review_item_info_tags/text()').extract())
                yield item

     next_page = response.xpath('//a[@id="review_next_page_link"]/@href')
     if next_page:
      url = response.urljoin(next_page[0].extract())
      yield scrapy.Request(url, self.parse_reviews)
3
  • For starters why use/import a crawlspider with ruls if your only going to defin your e if you arent going use... rules lol... I think youd be, ill answer you in the morning!
    – scriptso
    Jun 19, 2017 at 6:25
  • @scriptso, yes please answer for i cannot understand you.
    – Hamza
    Jun 19, 2017 at 7:50
  • Please post the full log too.
    – Kaushik NP
    Jun 19, 2017 at 14:53

2 Answers 2

8

I like to point out that in your question you speak of a website booking.com but in your spider you have the the links to the website of which are the official documents for scrapy's tutorials... Will continue to use the quotes site for the sake of explanation ....

Okay, here we go... So in your code snippet you are using a crawl spider, of which is worth mentioning that the parse function is already a part of the logic behind the Crawl spider. Like I mentioned earlier, by renaming your parse to different name such as parse_item which is the default initial function when you create the scroll spider but truthfully you can name it whatever you want. By doing so I believe I should actually crawl the site but it's all depends on your code being correct.

Screenshot of official Doc: Crawl Spider uses parse function as logic

In a nutshell, the difference between a generic spider and they crawl spider is that when using the crawl spider you use modules such as link extractor and rules of which set certain parameters so that when the start URL follows the pattern of which is used to navigate through the page, with various helpful argument to do just that... Of which the last rule set is the one with the car back to which you polish them. iIn other words... crawl spider creates the logic for request to navigate as desired.

enter image description here

Notice that inthe rules set.... I enter ... "/page." .... using "." is a regular expression that says.... "From the page im in... anyt links on this page that follow the pattern ..../page" it will follow AND callback to parse_item..."

This is A SUPER simple example ... as you can enter the patter to JUST follow or JUST callback to you item parse function...

with normal spider you manuallyhave to workout the site navigation to get the desired content you wish...

Example generic spider


CrawlSpider

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from quotes.items import QuotesItem

class QcrawlSpider(CrawlSpider):
    name = 'qCrawl'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']

    rules = (
        Rule(LinkExtractor(allow=r'page/.*'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        item = QuotesItem()
        item['quote'] =response.css('span.text::text').extract()
        item['author'] = response.css('small.author::text').extract()
        yield item

Generic Spider

import scrapy
from quotes.items import QuotesItem

class QspiSpider(scrapy.Spider):
    name = "qSpi"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ['http://quotes.toscrape.com']

    def parse(self, response):
        for quote in response.css("div.quote"):
            item = QuotesItem()
            item['quote'] = quote.css('span.text::text').extract()
            item['author'] = quote.css('small.author::text').extract()
            item['tags'] = quote.css("div.tags > a.tag::text").extract()
            yield item

        for nextPage in response.css('li.next a::attr(href)').extract():
            yield scrapy.Request(response.urljoin(nextPage))

a


EDIT: Additional info at request of OP

"...I cannot understand how to add arguments to the Rule parameters"

Okay... lets look at the official documentation just to reiterate the crawl spiders definition...

enter image description here

So crawl spiders create the logic behind following links by using the rule set... now lets say I want to crawl craigslist with a crawl spider for only house hold items for sale.... I want you to take notice of the to things in red....

enter image description here

For number one is to show that when im on craigslist house hold items page

SO we gather that ... anything in "search/hsh..." will be pages for house hold items list, from the first page from the lading page.

For the big red number "2"... is to show that when we are in the actual items posted... all items seem to have ".../hsh/..." so that any links inside the previs page that has this patter I want to follow and scrape from there ... SO my spider would be something like ...

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from craigListCrawl.items import CraiglistcrawlItem

class CcrawlexSpider(CrawlSpider):
    name = 'cCrawlEx'
    allowed_domains = ['columbia.craigslist.org']
    start_urls = ['https://columbia.craigslist.org/']

    rules = (
        Rule(LinkExtractor(allow=r'search/hsa.*'), follow=True),
        Rule(LinkExtractor(allow=r'hsh.*'), callback='parse_item'),
    )

    def parse_item(self, response):
        item = CraiglistcrawlItem()
        item['title'] = response.css('title::text').extract()
        item['description'] = response.xpath("//meta[@property='og:description']/@content").extract()
        item['followLink'] = response.xpath("//meta[@property='og:url']/@content").extract()
        yield item

I want you to think of it like steps you take to get from the landingpage to where your page with the content is... So we landed on the page which is our start_url... tSo the we said that the House Hold Items has a patter so As you can see for the first rule...

Rule(LinkExtractor(allow=r'search/hsa.*'), follow=True)

Here it says allow the regular expression patter "search/hsa." be followed ... remember that "." is a regular expression that is to match anything after "search/hsa"in this case atleast.

So the logic continues and then say that any link with the pattern "hsh.*" is to be calledback to my parse_item

If you think of it as steps from on page to an other as far as "clicks" it takes it should help... though perfectly acceptable, generic spiders will give you the most control as far as resources your scrapy project will end upusing meaning that a well written spider should be more precise and far faster.

4
  • Thanks for such a detailed answer. I was finding it hard to understand, your answer helped me a lot. Can you please help me more? I cannot understand how to add arguments to the Rule parameters.
    – Hamza
    Jun 20, 2017 at 12:31
  • 1
    Added to my respone hope I helped some
    – scriptso
    Jun 20, 2017 at 20:25
  • Thanks! Sir, for such a detailed response, it has helped me on my way to using scrapy. You are right generic spiders give much more control and features than crawl spiders, so I am using generic spiders instead of crawls. I have learned quite much but it's still insufficient. Right Now I am struck at scraing ajax loaded contents using scrapy. I have posted a question about it , if you can, please help me here too. Thanks! [the link] stackoverflow.com/questions/44856174/…
    – Hamza
    Jul 1, 2017 at 0:05
  • This question is obviously very old but @scriptso this answer is freaking awesome and just pulled me out of 3 day wall I've been up against, thank you. Jul 29, 2020 at 12:59
0

You are overriding parse method on a CrawlSpider subclass which is not recommended as per documentation:

When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work.

Though, I don't see Rules in your Spider so I'd recommend just switching to scrapy.spiders.Spider instead of scrapy.spiders.CrawlSpider. Just inherit from Spider class and run it again, it should work as you expect.

2
  • Don't mind, i am still new to python and scrapy. Can you please tell me how not to override parse. Can you please show me. Thanks
    – Hamza
    Jun 19, 2017 at 7:55
  • Scrappy crawl spider as the parse function as a function so you would just rename your parse to parse_{a given name...} doc.scrapy.org/en/latest/topics/spiders.html#crawlspider
    – scriptso
    Jun 19, 2017 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.