3

I've set up my PATH variables and I think I'm configuring everything right. But when I run the "scrapy crawl dmoz" within my startproject folder, I get these error messages:

c:\matt\testing\dmoz>scrapy crawl dmoz
2012-04-24 18:12:56-0400 [scrapy] INFO: Scrapy 0.14.0.2841 started (bot: dmoz)
2012-04-24 18:12:56-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole,         
CloseSpider, WebService, CoreStats, SpiderState
2012-04-24 18:12:56-0400 [scrapy] DEBUG: Enabled downloader middlewares:    
HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware,
faultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware,   
HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2012-04-24 18:12:56-0400 [scrapy] DEBUG: Enabled spider middlewares:   
HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware,  DepthMiddware
2012-04-24 18:12:56-0400 [scrapy] DEBUG: Enabled item pipelines:
Traceback (most recent call last):
File "c:\Python27\Scripts\scrapy", line 4, in <module>
execute()
File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7- 
win32.egg\scrapy\cmdline.py", line 132, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7-
win32.egg\scrapy\cmdline.py", line 97, in _run_print_help
func(*a, **kw)
File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7-
win32.egg\scrapy\cmdline.py", line 139, in _run_command
cmd.run(args, opts)
File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7-
win32.egg\scrapy\commands\crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7-  
win32.egg\scrapy\spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: dmoz'

Does anyone know what might be going on?

3
  • 2
    Show the directory structure you have, the source your spider class and of settings module
    – warvariuc
    Apr 25, 2012 at 7:27
  • choose a best answer please !
    – realtebo
    Jun 8, 2013 at 10:17
  • Also, the tutorial fails to mention that you ALSO need this library: github.com/scrapy/queuelib Nov 19, 2013 at 2:25

2 Answers 2

12

I had this issue too.

It is because the scrapy tutorial asks you to place the spider you create in /dmoz/spiders/ but scrapy is looking in tutorial/tutorial/spiders.

Save the dmoz_spider.py in tutorial/tutorial/spiders and the crawl should work.

0
0

try in a command line

C:\Users\Akhtar Wahab> python # any any directory path

if it works

try

scrapy version

if its also working

then make sure you made scrapy project with

scrapy startproject name

if above everything is Positive for you

then make sure you are Running scraper command in a directory where scrapy.cfg is present .

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.