2

I'm learning how to work with scrapy + splash. I've created a project inside with a virtual environment and I'm now doing this tutorial: https://github.com/scrapy-plugins/scrapy-splash.

I've ran splash with:

$ docker run -p 8050:8050 scrapinghub/splash

which resulted in:

2017-01-12 09:18:50+0000 [-] Log opened.
2017-01-12 09:18:50.225754 [-] Splash version: 2.3
2017-01-12 09:18:50.227033 [-] Qt 5.5.1, PyQt 5.5.1, WebKit 538.1, sip 4.17, Twisted 16.1.1, Lua 5.2
2017-01-12 09:18:50.227201 [-] Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4]
2017-01-12 09:18:50.227645 [-] Open files limit: 1048576
2017-01-12 09:18:50.227882 [-] Can't bump open files limit
2017-01-12 09:18:50.333978 [-] Xvfb is started: ['Xvfb', ':1', '-screen', '0', '1024x768x24']
2017-01-12 09:18:50.438528 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2017-01-12 09:18:50.597573 [-] verbosity=1
2017-01-12 09:18:50.597747 [-] slots=50
2017-01-12 09:18:50.597820 [-] argument_cache_max_entries=500
2017-01-12 09:18:50.598696 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2017-01-12 09:18:50.601924 [-] Site starting on 8050
2017-01-12 09:18:50.602119 [-] Starting factory <twisted.web.server.Site object at 0x7ff528490be0>

When I run the following spider:

import scrapy
from scrapy_splash import SplashRequest
class MySpider(scrapy.Spider):
    name = 'spiderman'
    domain = ['web']
    start_urls = ['http://www.example.com']

    def parse(self, response):
        print(response.body)

Everything works fine; scrapy returns the body html. However, when I try SplashRequest from the tutorial like this:

import scrapy
from scrapy_splash import SplashRequest
class MySpider(scrapy.Spider):
    name = 'spiderman'
    domain = ['web']
    start_urls = ['http://www.example.com']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse,
                            args = {'wait':0.5},)

    def parse(self, response):
        response.body

I get the following messages in my terminal:

File "/Users/username/myVirtualEnvironment/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 61: Connection refused.
2017-01-12 11:02:50 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-12 11:03:06 [scrapy.downloadermiddlewares.retry] DEBUG: 
Retrying <GET http://192.168.59.103:8050/robots.txt> (failed 1 times): TCP connection timed out: 60: Operation timed out

My guess is that splash causes some connection issues, but I don't know how to fix them. I've added:

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.7'
DOWNLOAD_DELAY = 0.25

But it doesn't help!

Q: Does anyone know how to solve this issue?

EDIT: Changing ROBOTSTXT_OBEY to False doesn't work. Entire console log:

$ scrapy crawl spiderman
2017-01-12 11:25:18 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: myScrapingProject)
2017-01-12 11:25:18 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myScrapingProject', 'DOWNLOAD_DELAY': 0.25, 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage', 'NEWSPIDER_MODULE': 'myScrapingProject.spiders', 'SPIDER_MODULES': ['myScrapingProject.spiders'], 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.7'}
2017-01-12 11:25:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2017-01-12 11:25:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-12 11:25:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-12 11:25:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-01-12 11:25:18 [scrapy.core.engine] INFO: Spider opened
2017-01-12 11:25:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-12 11:25:18 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-12 11:26:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-12 11:26:33 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.example.com via http://192.168.59.103:8050/render.html> (failed 1 times): TCP connection timed out: 60: Operation timed out.
2017-01-12 11:27:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-12 11:27:48 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.example.com via http://192.168.59.103:8050/render.html> (failed 2 times): TCP connection timed out: 60: Operation timed out.
2017-01-12 11:28:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-12 11:29:03 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.example.com via http://192.168.59.103:8050/render.html> (failed 3 times): TCP connection timed out: 60: Operation timed out.
2017-01-12 11:29:03 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.example.com via http://192.168.59.103:8050/render.html>
Traceback (most recent call last):
  File "/Users/username/myVirtualEnvironment/lib/python3.6/site-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Users/username/myVirtualEnvironment/lib/python3.6/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Users/username/myVirtualEnvironment/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.internet.error.TCPTimedOutError: TCP connection timed out: 60: Operation timed out.
2017-01-12 11:29:03 [scrapy.core.engine] INFO: Closing spider (finished)
2017-01-12 11:29:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
 'downloader/exception_type_count/twisted.internet.error.TCPTimedOutError': 3,
 'downloader/request_bytes': 1746,
 'downloader/request_count': 3,
 'downloader/request_method_count/POST': 3,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 1, 12, 10, 29, 3, 935527),
 'log_count/DEBUG': 4,
 'log_count/ERROR': 1,
 'log_count/INFO': 10,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'splash/render.html/request_count': 1,
 'start_time': datetime.datetime(2017, 1, 12, 10, 25, 18, 451764)}
2017-01-12 11:29:03 [scrapy.core.engine] INFO: Spider closed (finished)

EDIT2: If I run curl http://localhost:8050/render.html?url=http%3A%2F%2Fwww.examp‌​le.com%2F in a new terminal window, I get the following ouput in the terminal window that I used to run splash with:

process 1: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/etc/machine-id": No such file or directory
See the manual page for dbus-uuidgen to correct this issue.
2017-01-12 10:48:03.341100 [events] {"path": "/render.html", "load": [0.07, 0.02, 0.0], "fds": 19, "client_ip": "172.17.0.1", "_id": 140690919672912, "method": "GET", "rendertime": 6.497595548629761, "active": 0, "qsize": 0, "maxrss": 83860, "args": {"uid": 140690919672912, "url": "http://www.examp\u200c\u200ble.com/"}, 
"timestamp": 1484218083, "status_code": 200, "user-agent": "curl/7.51.0"}
2017-01-12 10:48:03.343167 [-] "172.17.0.1" - - [12/Jan/2017:10:48:02 +0000] "GET /render.html?url=http%3A%2F%2Fwww.examp\xe2\x80\x8c\xe2\x80\x8ble.com%2F HTTP/1.1" 200 1262 "-" "curl/7.51.0"
7
  • Have you installed scrapy_splash.SplashMiddleware in your Scrapy project settings in DOWNLOADER_MIDDLEWARES like described in the README? You can also disable robots.txt handling with ROBOTSTXT_OBEY=False in your scrapy settings.py. You can also check that Splash is up and running by opening the web interface at localhost:8050 Jan 12, 2017 at 10:19
  • Yeah typo sorry. Ehm, how do I install scrapy_splash.SplashMiddleware, I can't find it in the readme file sorry. I think it's installed since I also get the following message: 2017-01-12 11:25:18 [scrapy.middleware] INFO: Enabled downloader middlewares: 'scrapy_splash.SplashCookiesMiddleware', 'scrapy_splash.SplashMiddleware', Yes splash is up and running!
    – titusAdam
    Jan 12, 2017 at 10:28
  • And have you tried disabling robots.txt handling with ROBOTSTXT_OBEY=False? If it still does not work, paste your console logs where you run scrapy crawl (all of it, not just the end with Retrying <GET http://192.168.59.103:8050/robots.txt>). And if you see anything in the Splash logs in that other console, you can paste that too. Jan 12, 2017 at 10:31
  • @paultrmbrth unfortunately ROBOTSTXT_OBEY=FALSE doesn't help. Edited; now with entire console log.
    – titusAdam
    Jan 12, 2017 at 10:40
  • Then I don't know. Can you use curl to test your setup? something like curl http://localhost:8050/render.html?url=http%3A%2F%2Fwww.example.com%2F Jan 12, 2017 at 10:45

2 Answers 2

8

The problem is that SPLASH_URL must point to the locally running Splash instance, usually running at http://localhost:8050,

and not the value used as an example in scrapy-splash README -- http://192.168.59.103:8050 which appear in the error logs:

Retrying <GET http://www.example.com via http://192.168.59.103:8050/render.html> (failed 1 times)

The OP tested curl http://localhost:8050/render.html?url=http%3A%2F%2Fwww.examp‌​‌​le.com%2F, which worked, so the settings should say:

SPLASH_URL = 'http://localhost:8050'
3
  • I still get the problem after changing SPLASH_URL into 'localhost:8050' ! are there any other possible reasons for this problem?
    – Jeff
    Jul 25, 2017 at 11:46
  • @user345, you may want to open another question on StackOverflow. Did you include 'http://' in your SPLASH_URL? Jul 25, 2017 at 11:57
  • Thanks for your answer which helped solve the problem. I wonder though why the scrapy-splash README is not clearer about it (or should it be obvious)?
    – gofvonx
    Jan 17, 2019 at 15:24
0

I had my splash app running in background , so I changed SPLASH_URL from the default on to the one on which my splash app was running , and it worked enter image description here

2
  • As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
    – Community Bot
    Oct 1, 2021 at 6:20
  • The accepted answer already contains this information. Why did you post this new answer?
    – wovano
    Oct 1, 2021 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.