7

Scrapy version: 1.0.5

I have searched for long time, but most of workarounds don't work in current Scrapy version.

My spider is defined in jingdong_spider.py, and the interface(learn it by Scrapy Documentation) to run spider is below:

# interface
def search(keyword):
    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
    runner = CrawlerRunner()
    d = runner.crawl(JingdongSpider,keyword)
    d.addBoth(lambda _: reactor.stop())
    reactor.run() # the script will block here until the crawling is finished

Then in temp.py I will call the search(keyword) above to run spider.

Now the problem: I called search(keyword) once, and it worked well.But I called it twice, for instance,

in temp.py

search('iphone')
search('ipad2')

it reported:

Traceback (most recent call last): File "C:/Users/jiahao/Desktop/code/bbt_climb_plus/temp.py", line 7, in search('ipad2') File "C:\Users\jiahao\Desktop\code\bbt_climb_plus\bbt_climb_plus\spiders\jingdong_spider.py", line 194, in search reactor.run() # the script will block here until the crawling is finished File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1193, in run self.startRunning(installSignalHandlers=installSignalHandlers) File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1173, in startRunning ReactorBase.startRunning(self) File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 684, in startRunning raise error.ReactorNotRestartable() twisted.internet.error.ReactorNotRestartable

The first search(keyword) succeeded, but the latter got wrong.

Could you help?

2 Answers 2

5

In your code sample you are making calls to twisted.reactor starting it on every function call. This is not working because there is only one reactor per process and you cannot start it twice.

There are two ways to solve your problem, both described in documentation here. Either stick with CrawlerRunner but move reactor.run() outside your search() function to ensure it is only called once. Or use CrawlerProcess and simply call crawler_process.start(). Second approach is easier, your code would look like this:

from scrapy.crawler import CrawlerProcess
from dirbot.spiders.dmoz import DmozSpider

def search(runner, keyword):
    return runner.crawl(DmozSpider, keyword)

runner = CrawlerProcess()
search(runner, "alfa")
search(runner, "beta")
runner.start()
3
  • 1
    Thanks. Your answer is awesome, but I need to use search(keyword) when I input something. i.e, search(keyword)need to be called when user do sth(like click button,etc).Such that search(keyword) can't hard-coded. At last, I solve my problem with multiprocessing.
    – guo
    Apr 5, 2016 at 9:06
  • 2
    I am afraid I haven't explain clearly. search(keyword) may be called twice, triple, four times... It depends.
    – guo
    Apr 5, 2016 at 9:08
  • got error.. processCrawl = CrawlerProcess(get_project_settings()) in some function, and when i called again it said twisted.internet.error.ReactorNotRestartable
    – Budi Mulyo
    Mar 14, 2019 at 4:12
5

As Pawel Miech said

In your code sample you are making calls to twisted.reactor starting it on every function call. This is not working because there is only one reactor per process and you cannot start it twice.

I found a way to solve my problem. Just using multiprocessing.

it will be like:

from multiprocessing import Process
def run_spider(keyword):
    if __name__ == '__main__':
        p = Process(target=jingdong_spider.search, args=(keyword.encode('utf-8'),))
        p.start()
        p.join()

If everyone has problem in using python-multiprocessing. Better look at python documentation.

2
  • Although it's a dangling workaround, it works well. You've saved my day! I had 50 minutes to make my project work. Apr 5, 2017 at 6:50
  • I got error.. p.start() File "/usr/local/lib/python3.6/multiprocessing/process.py", line 103, in start 'daemonic processes are not allowed to have children'
    – Budi Mulyo
    Mar 14, 2019 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.