0

I'm using scrapy in order to fetch some web content, in this fashion:

class PitchforkTracks(scrapy.Spider):
    name = "pitchfork_tracks"
    allowed_domains = ["pitchfork.com"]
    start_urls = [
                    "http://pitchfork.com/reviews/best/tracks/?page=1",
                    "http://pitchfork.com/reviews/best/tracks/?page=2",
                    "http://pitchfork.com/reviews/best/tracks/?page=3",
     ]

everything is working fine.

now, instead of hitting the pages directely, I would like to scrape google caches of the same pages.

what is the proper syntaxto achieve that?

PS: I have tried "cache:http://pitchfork.com/reviews/best/tracks/?page=1",, to no avail.

1 Answer 1

3

you can use following Google URL for scraping cache page

http://webcache.googleusercontent.com/search?q=cache:http://pitchfork.com/reviews/best/tracks/?page=1

1
  • does webcache.googleusercontent.com block bots ? Aug 3 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.