9

I have installed Scrapy,and import it in python, every things look fine. But it cause error when i try a example in http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/tutorial.html.

I run scrapy crawl swspider, then i get:

> 2018-05-14 14:24:16 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: tutorial)
> 2018-05-14 14:24:16 [scrapy.utils.log] INFO: Versions: lxml 3.2.1.0,
> libxml2 2.9.1, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted
> 18.4.0, Python 2.7.5 (default, Nov 20 2015, 02:00:19) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)], pyOpenSSL 0.13.1 (OpenSSL 1.0.1e-fips 11
> Feb 2013), cryptography 0.8.2, Platform
> Linux-3.10.0-327.el7.x86_64-x86_64-with-centos-7.2.1511-Core
> 2018-05-14 14:24:16 [scrapy.crawler] INFO: Overridden settings:
> {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES':
> ['tutorial.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'tutorial'}
> Traceback (most recent call last):   File
> "/disk1/wulixin/install/bin/scrapy", line 11, in <module>
>     sys.exit(execute())   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 150, in execute
>     _run_print_help(parser, _run_command, cmd, args, opts)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 90, in _run_print_help
>     func(*a, **kw)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 157, in _run_command
>     cmd.run(args, opts)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/commands/crawl.py",
> line 57, in run
>     self.crawler_process.crawl(spname, **opts.spargs)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 170, in crawl
>     crawler = self.create_crawler(crawler_or_spidercls)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 198, in create_crawler
>     return self._create_crawler(crawler_or_spidercls)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 203, in _create_crawler
>     return Crawler(spidercls, self.settings)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 55, in __init__
>     self.extensions = ExtensionManager.from_crawler(self)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/middleware.py",
> line 58, in from_crawler
>     return cls.from_settings(crawler.settings, crawler)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/middleware.py",
> line 34, in from_settings
>     mwcls = load_object(clspath)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/utils/misc.py",
> line 44, in load_object
>     mod = import_module(module)   File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in
> import_module
>     __import__(name)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/extensions/memusage.py",
> line 16, in <module>
>     from scrapy.mail import MailSender   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/mail.py",
> line 25, in <module>
>     from twisted.internet import defer, reactor, ssl   File "/disk1/wulixin/install/lib64/python2.7/site-packages/twisted/internet/ssl.py",
> line 230, in <module>
>     from twisted.internet._sslverify import (   File "/disk1/wulixin/install/lib64/python2.7/site-packages/twisted/internet/_sslverify.py",
> line 15, in <module>
>     from OpenSSL._util import lib as pyOpenSSLlib ImportError: No module named _util
1
  • There is something very wrong going on here - possibly some kind of version mismatch. Try to install scrapy in a virtual env, together with all the dependencies, and see if that works.
    – bosnjak
    May 14, 2018 at 21:30

5 Answers 5

13

you need upgrade pyopenssl

sudo pip install pyopenssl --user --upgrade
3

If you are using the newest version of Twisted. You can try downgrade Twisted:

pip install Twisted==16.4.1
0

You are not activating the scrapy environment

$source activate ScrapyEnvironment
0

For some mac os x user, this might not solve the problem since the older version of pyOpenSSL never get uninstall. The way to solve it is to download the gz file from PyPI a link. Then unzip the compressed file. Navigate into it and then run the following command

sudo python setup.py install

Then pyOpenSSL is correctly updated.

0

This is the version problem and use the code to update:

pip install --upgrade --ignore-installed pyopenssl

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.