0

I am trying to create a spider with the package "Scrapy" that gets a lists of URLs and crawls them. I have searched stackoverflow for an answer but could not find something that will solve the issue.

My script is as follows:

class Try(scrapy.Spider):
   name = "Try"

   def __init__(self, *args, **kwargs):
      super(Try, self).__init__(*args, **kwargs)
      self.start_urls = kwargs.get( "urls" )
      print( self.start_urls )

   def start_requests(self):
      print( self.start_urls )
      for url in self.start_urls:
          yield Request( url , self.parse )

   def parse(self, response):
      d = response.xpath( "//body" ).extract()

When I crawl the spider:

Spider = Try(urls = [r"https://www.example.com"])
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(Spider)
process.start()

I get the following info printed while printing self.start_urls:

  • In the __init__ function printed on screen is: [r"https://www.example.com"] (as passed to the spider).
  • In the start_requests function printed on screen is: None

Why do I get None? Is there another way to approach this issue? or Is there any mistakes in my spider's class?

Thanks for any help given!

2
  • use other name for list to keep urls at start - self.start_urls is used by scrapy so it can remove them. I'm wonder if you put urls in self.start_urls in __init__ then maybe it will use them and you don't have to use start_requests ?
    – furas
    Dec 31, 2017 at 14:50
  • process.crawl will create a new Try object without parameters when it calls from_crawler. See Crawler class in Scrapy Source Code. github.com/scrapy/scrapy/blob/master/scrapy/…
    – matiskay
    Dec 31, 2017 at 15:35

2 Answers 2

2

I would suggest to use the Spider Class in process.crawl and pass urls parameters there.

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import Request


class Try(scrapy.Spider):
   name = 'Try'

   def __init__(self, *args, **kwargs):
      super(Try, self).__init__(*args, **kwargs)
      self.start_urls = kwargs.get("urls")

   def start_requests(self):
      for url in self.start_urls:
          yield Request( url , self.parse )

   def parse(self, response):
      d = response.xpath("//body").extract()

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(Try, urls=[r'https://www.example.com'])
process.start()
0

If I run

process.crawl(Try, urls=[r"https://www.example.com"])

then it send urls to Try as I expect. And even I don't need start_requests.

import scrapy

class Try(scrapy.Spider):

   name = "Try"

   def __init__(self, *args, **kwargs):
       super(Try, self).__init__(*args, **kwargs)
       self.start_urls = kwargs.get("urls")

   def parse(self, response):
       print('>>> url:', response.url)
       d = response.xpath( "//body" ).extract()

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(Try, urls=[r"https://www.example.com"])
process.start()

But if I use

spider = Try(urls = ["https://www.example.com"])

process.crawl(spider)

then it looks like it runs new Try without urls and then list is empty.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.