I'm using scrapy to crawl some pages. I am using python 2.7. The spider returns the response object, and I am inspecting URLs found on the page. I want to restrict the spider to only follow URLS that are subpaths of a location I specify.
For instance, I would like to specify that the spider should only follows links below www.google.com/policies/privacy/
The links extracted from the response object follow a number of different conventions.
E.g.
- ../../policies/privacy/
- ../../policies/privacy/example/collect-information.html
- #infocollect
- /intl/en_uk/policies/privacy/google_privacy_policy_en_uk.pdf
- //myaccount.google.com/
- https://support.google.com/policies/troubleshooter/2990837?hl=en-GB
I can't work out how to do it. I have just used a simple find method on the string. It's not robust or that clever it seems to me.
import scrapy
class googleSpider(scrapy.Spider):
name = "google"
allowed_domains = ["google.co.uk"]
start_urls = [
"http://www.google.co.uk/intl/en/policies/privacy/"
]
def parse(self, response):
for href in response.xpath('//a/@href').extract():
if href.find('/policies/privacy/') != -1:
yield scrapy.Request(response.urljoin(href), callback=self.parse_dir_contents)
def parse_dir_contents(self, response):
pass