29

I used to scrape e-commerce webpage occasionally to get product prices information. I have not used the scraper built using Scrapy in a while and yesterday was trying to use it - I run into a problem with bot protection.

It is using CloudFlare’s DDOS protection which is basically using JavaScript evaluation to filter out the browsers (and therefore scrapers) with JS disabled. Once the function is evaluated, the response with calculated number is generated. In return, service sends back two authentication cookies which attached to each request allow to normally crawl the site. Here's the description of how it works.

I have also found a cloudflare-scrape Python module that uses external JS evaluation engine to calculate the number and send the request back to server. I'm not sure how to integrate it into Scrapy though. Or maybe there's a smarter way without using JS execution? In the end, it's a form...

I'd apriciate any help.

4
  • what's wrong with executing the JS? a lot of content can't be reached without it...
    – dandavis
    Oct 20, 2015 at 22:20
  • @dandavis Scrapy doesn't do JS execution.
    – Rejected
    Oct 21, 2015 at 5:42
  • without JS, scraping is getting more useless all the time. you should consider using a browser to scrape. my scrape scripts can get behind logins, grab dynamic and paginated content, even fake-click to request more dynamic sub-content before scraping, and i don't think any have been more than 10 lines of code or needed a big external library. all you need is tampermonkey and a download routine... KISS. if you want it more automated, you can then switch the userscript to phantomJS or something like that.
    – dandavis
    Oct 21, 2015 at 20:48
  • for executing JS you could use splash which is totally integrated with scrapy.
    – eLRuLL
    Oct 28, 2015 at 20:09

3 Answers 3

21

So I executed JavaScript using Python with help of cloudflare-scrape.

To your scraper, you need to add the following code:

def start_requests(self):
  for url in self.start_urls:
    token, agent = cfscrape.get_tokens(url, 'Your prefarable user agent, _optional_')
    yield Request(url=url, cookies=token, headers={'User-Agent': agent})

alongside parsing functions. And that's it!

Of course, you need to install cloudflare-scrape first and import it to your spider. You also need a JS execution engine installed. I had Node.JS already, no complaints.

6
  • Hi, I tried using your solution for a website using cloudflare - It seems like the right request is being sent, but I'm still timing out with 503 errors from the first request. Here's what the reuqests look like: DEBUG: "GET /cdn-cgi/l/chk_jschl?jschl_answer=386&jschl_vc=3ee197400dbcca30577bcb3949cf781f&pass=1446694346.755-esyN0f4s4%2F HTTP/1.1" 302 165 DEBUG: "GET /forumdisplay.php?29-DotA-Chat HTTP/1.1" 200 21367 2015-11-04 22:32:48 [scrapy] DEBUG: Gave up retrying <GET http://nadota.com/forumdisplay.php?29-DotA-Chat> (failed 3 times): 503 Service Unavailable
    – ddnm
    Nov 5, 2015 at 3:36
  • @ddnm, I updated the code. It will work now. Change cookies={"cfuid":token["cfuid"]} to cookies=token Nov 21, 2015 at 3:25
  • I get Error parsing Cloudflare IUAM Javascript challenge as per March 2019
    – Umair Ayub
    Mar 21, 2019 at 11:14
  • 4
    cfscrape no longer works and no longer maintained, as per: github.com/Anorov/cloudflare-scrape/issues/347 Apr 18, 2021 at 9:39
  • 2
    this library is not working (28 april 2021). Probably developer not interested. ValueError: Unable to identify Cloudflare IUAM Javascript on website. Cloudflare may have changed their technique, or there may be a bug in the script. Apr 28, 2021 at 8:43
8

Obviously the best way to do this would be to whitelist your IP in CloudFlare; if this isn't suitable let me recommend the cloudflare-scrape library. You can use this to get the cookie token, then provide this cookie token in your Scrapy request back to the server.

-3

If you're getting 503 Error you can follow these guidelines:

  1. Go to settings.py
  2. Search for: USER_AGENT
  3. Here you will see the default bot user agent by scrapy. Replace that default with this:
    USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
    

Not the answer you're looking for? Browse other questions tagged or ask your own question.