2

I have been trying to hone my python skills by building scrapers and recently switched from bs4 to scrapy so that I can use its multithreading and download delay features. I have been able to make a basic scraper and output the data to csv, but when I try to add a recursive feature I run into problems. I tried following the advice from Scrapy Recursive download of Content but keep getting the following error:

DEBUG: Retrying http://medford.craigslist.org%20%5Bu'/cto/4359874426.html'%5D> DNS lookup failed: address not found

This makes me think the way I am trying to join the links isn't work as it's inserting characters into the url, but I can't figure out how to fix it. Any advice?

Here's the code:

#-------------------------------------------------------------------------------
# Name:        module1
# Purpose:
#
# Author:      CD
#
# Created:     02/03/2014
# Copyright:   (c) CD 2014
# Licence:     <your licence>
#-------------------------------------------------------------------------------
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.http import Request
from scrapy.selector import *

class PageSpider(BaseSpider):
    name = "cto"

    start_urls = ["http://medford.craigslist.org/cto/"]

    rules = (Rule(SgmlLinkExtractor(allow=("index\d00\.html", ), restrict_xpaths=('//p[@class="nextpage"]' ,))
        , callback="parse", follow=True), )

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//span[@class='pl']")
        for titles in titles:
            item = CraigslistSampleItem()
            item['title'] = titles.select("a/text()").extract()
            item['link'] = titles.select("a/@href").extract()

            url = "http://medford.craiglist.org %s" % item['link']
            yield Request(url=url, meta={'item': item}, callback=self.parse_item_page)

    def parse_item_page(self, response):
        hxs = HtmlXPathSelector(response)

        item = response.meta['item']
        item['description'] = hxs.select('//section[@id="postingbody"]/text()').extract()
        return item
5
  • url = "http://medford.craiglist.org %s" % item['link'] can't possibly be right, but I don't know what you actually wanted. You might like to know about the standard-library urlparse module.
    – zwol
    Mar 4, 2014 at 20:32
  • I pulled that line from the link I posted. What I was trying to do is have it pull the link it found and add that to the Craigslist url to get the address for each page. Mar 4, 2014 at 20:35
  • Insert import sys at the top of the file and sys.stderr.write(repr(item[link]) + "\n") immediately before the url = line. What does it print?
    – zwol
    Mar 4, 2014 at 20:48
  • Also, probably not your immediate problem, but for titles in titles: looks like a typo to me. Shouldn't it be for title in titles or similar?
    – zwol
    Mar 4, 2014 at 20:50
  • I thought the titles in titles was wrong too, but was following this tutorial - mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy I tried the code and got a list of links returned and then it started throwing that error. Tried the suggestion below as that was the format I had in my head, just couldn't remember the proper way to do it and it worked. I went through the codecademy and learnpythonthehardway courses, and am running into questions that the documentation would probably understand but it's still a little over my head. Any suggestions for learning the nuances more? Mar 4, 2014 at 20:57

1 Answer 1

2

Turns out your code:

 url = "http://medford.craiglist.org %s" % item['link']

generates:

http://medford.craigslist.org [u'/cto/4359874426.html']

The item['link'] returns a list in your code and not a string as you are expecting it to. You need to do this:

url = 'http://medford.craiglist.org{}'.format(''.join(item['link']))
1
  • That worked! Thank you for explaining WHY I was having the error and not just how to fix it. That makes sense and the .join was what I thought I needed I just couldn't make it work before attempting what I originally posted. If only I could give extra points... One more question if you're able to help, since Craigslist has pages of 100 listings, how would I implement a recursive scrape of the links on the first page and then do the same for the following pages? For example, first page is medford.craigslist.org/cto/ and then second page is medford.craigslist.org/cto/index100.html, etc Mar 4, 2014 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.