I try to create my first spider scraper using scrapy I use Dmoz as test, I get an error message: TypeError: Request url must be str or unicode, got NoneType But in the Debug I can see the right url
Code:
import scrapy
import urlparse
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = ["http://www.dmoz.org/search?q=france&all=no&t=regional&cat=all"]
def parse(self, response):
sites = response.css('#site-list-content > div.site-item > div.title-and-desc')
for site in sites:
yield {
'name': site.css('a > div.site-title::text').extract_first().strip(),
'url': site.xpath('a/@href').extract_first().strip(),
'description': site.css('div.site-descr::text').extract_first().strip(),
}
nxt = response.css('#subcategories-div > div.previous-next > div.next-page')
next_page = nxt.css('a::attr(href)').extract_first()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
Logs:
2016-10-18 11:17:03 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/search?q=france&start=20&type=next&all=no&t=regional&cat=all> (referer: http://www.dmoz.org/search?q=france&all=no&t=regional&cat=all)
2016-10-18 11:17:03 [scrapy] ERROR: Spider error processing <GET http://www.dmoz.org/search?q=france&start=20&type=next&all=no&t=regional&cat=all> (referer: http://www.dmoz.org/search?q=france&all=no&t=regional&cat=all)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback
yield next(it)
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "/ENV/bin/tutorial/dirbot/spiders/dmoz.py", line 25, in parse
yield scrapy.Request(next_page, callback=self.parse)
File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 25, in __init__
self._set_url(url)
File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 51, in _set_url
raise TypeError('Request url must be str or unicode, got %s:' % type(url).__name__)
TypeError: Request url must be str or unicode, got NoneType:
2016-10-18 11:17:03 [scrapy] INFO: Closing spider (finished)
2016-10-18 11:17:03 [scrapy] INFO: Stored json feed (20 items) in: test.json
2016-10-18 11:17:03 [scrapy] INFO: Dumping Scrapy stats:
yield scrapy.Request(next_page, callback=self.parse)
needs to be inside the if, you checkif next_page is not None
but still make a Request regardless2016-10-18 11:45:44 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/search?q=france&start=20&type=next&all=no&t=regional&cat=all> (referer: http://www.dmoz.org/search?q=france&all=no&t=regional&cat=all) 2016-10-18 11:45:44 [scrapy] INFO: Closing spider (finished)
next_page
is probably never finding anything, have you verified that it ever finds anything?