Dismiss
Announcing Stack Overflow Documentation

We started with Q&A. Technical documentation is next, and we need your help.

Whether you're a beginner or an experienced developer, you can contribute.

Sign up and start helping → Learn more about Documentation →

I'm using scrapy to crawl some pages. I am using python 2.7. The spider returns the response object, and I am inspecting URLs found on the page. I want to restrict the spider to only follow URLS that are subpaths of a location I specify.

For instance, I would like to specify that the spider should only follows links below www.google.com/policies/privacy/

The links extracted from the response object follow a number of different conventions.

E.g.

I can't work out how to do it. I have just used a simple find method on the string. It's not robust or that clever it seems to me.

import scrapy

class googleSpider(scrapy.Spider):
    name = "google"
    allowed_domains = ["google.co.uk"]
    start_urls = [
        "http://www.google.co.uk/intl/en/policies/privacy/"
    ]

    def parse(self, response):
        for href in response.xpath('//a/@href').extract():
            if href.find('/policies/privacy/') != -1:
                yield scrapy.Request(response.urljoin(href), callback=self.parse_dir_contents)

    def parse_dir_contents(self, response):
        pass
share|improve this question
    
Can you share the Scrapy code you tried so far? – paul trmbrth Jun 22 at 11:40
    
Sure. You can see I'm not tackling the issue properly. – Donal Mee Jun 22 at 13:57
up vote 0 down vote accepted

You could use a LinkExtractor for this. The default one normalizes the links.

Then, it's a matter of building Requests out of the links you get from .extract_links(response)

Check this scrapy shell example:

$ scrapy shell https://www.google.com/policies/privacy/
2016-06-22 18:03:19 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot)
(...edited...)
2016-06-22 18:03:20 [scrapy] DEBUG: Crawled (200) <GET https://www.google.com/policies/privacy/> (referer: None)
(...edited...)

>>> from scrapy.linkextractors import LinkExtractor
>>> for l in LinkExtractor().extract_links(response):
...     print(l.url)
... 
https://www.google.com/
(...edited...)
https://support.google.com/accounts/answer/32046?hl=en
https://www.google.com/trends/
https://www.youtube.com/trendsmap
https://privacy.google.com/?hl=en
https://www.google.com/policies/technologies/location-data/
https://www.google.com/policies/technologies/wallet/
https://www.google.com/policies/technologies/voice/
https://www.google.com/safetycenter/families/start/
https://www.google.com/intl/en/about/
https://www.google.com/intl/en/policies/privacy/
https://www.google.com/intl/en/policies/terms/

>>> for l in LinkExtractor().extract_links(response):
...     if response.url in l.url:
...         print(l.url)
... 
https://www.google.com/policies/privacy/
https://www.google.com/policies/privacy/frameworks/
https://www.google.com/policies/privacy/key-terms/
https://www.google.com/policies/privacy/partners/
https://www.google.com/policies/privacy/archive/
https://www.google.com/policies/privacy/example/more-relevant-search-results.html
https://www.google.com/policies/privacy/example/connect-with-people.html
https://www.google.com/policies/privacy/example/sharing-with-others.html
https://www.google.com/policies/privacy/example/ads-youll-find-most-useful.html
https://www.google.com/policies/privacy/example/the-people-who-matter-most.html
https://www.google.com/policies/privacy/example/credit-card.html
https://www.google.com/policies/privacy/example/collect-information.html
https://www.google.com/policies/privacy/example/view-and-interact-with-our-ads.html
https://www.google.com/policies/privacy/example/device-specific-information.html
https://www.google.com/policies/privacy/example/device-identifiers.html
https://www.google.com/policies/privacy/example/phone-number.html
https://www.google.com/policies/privacy/example/may-collect-and-process-information.html
https://www.google.com/policies/privacy/example/sensors.html
https://www.google.com/policies/privacy/example/wifi-access-points-and-cell-towers.html
https://www.google.com/policies/privacy/example/our-partners.html
https://www.google.com/policies/privacy/example/advertising-services.html
https://www.google.com/policies/privacy/example/linked-with-information-about-visits-to-multiple-sites.html
https://www.google.com/policies/privacy/example/provide-services.html
https://www.google.com/policies/privacy/example/maintain-services.html
https://www.google.com/policies/privacy/example/protect-services.html
https://www.google.com/policies/privacy/example/develop-new-ones.html
https://www.google.com/policies/privacy/example/protect-google-and-our-users.html
https://www.google.com/policies/privacy/example/limit-sharing-or-visibility-settings.html
https://www.google.com/policies/privacy/example/improve-your-user-experience.html
https://www.google.com/policies/privacy/example/combine-personal-information.html
https://www.google.com/policies/privacy/example/to-make-it-easier-to-share.html
https://www.google.com/policies/privacy/example/may-not-function-properly.html
https://www.google.com/policies/privacy/example/sharing.html
https://www.google.com/policies/privacy/example/removing-your-content.html
https://www.google.com/policies/privacy/example/access-to-your-personal-information.html
https://www.google.com/policies/privacy/example/legal-process.html
https://www.google.com/policies/privacy/example/we-may-share.html
https://www.google.com/policies/privacy/example/to-show-trends.html
share|improve this answer
    
Hey paul, thanks a lot. That really solved my problem. I knew there must be an inbuilt method to deal with links. Thanks a lot. – Donal Mee Jun 22 at 16:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.