7

I am currently part of a team developing an application which includes a front end client.

Through this client we send the user data, each user has a user-id and the client talks to our server through a RESTful API asking the server for data.

For example, let's say we have a database of books, and the user can get the last 3 books an author wrote. We value our users' time and we would like users to be able to start using the product without explicit registration.

We value our database, we use our own proprietary software to populate it and would like to protect it as much as we can.

So basically the question is:

What can we do to protect ourselves from web scraping?

I would very much like to learn about some techniques to protect our data, we would like to prevent users from typing every single author name in the author search panel and fetching out the top three books every author wrote.

Any suggested reading would be appreciated.

I'd just like to mention we're aware of captchas and would like to avoid them as much as possible

3
  • set a daily limit of requests from same IP?
    – Raptor
    Jan 17, 2013 at 10:47
  • To use the same example, Some libraries might all use the same IP we would like to allow all the uses in the library to be able to use the program at once. Jan 17, 2013 at 10:48
  • 2
    Note that data mining (when used correctly) refers to the analysis, not the scraping of data from web sites. Jan 17, 2013 at 16:41

2 Answers 2

5

The main strategies for preventing this are:

  • require registration, so you can limit the requests per user
  • captchas for registration and non-registered users
  • rate limiting for IPs
  • require JavaScript - writing a scraper that can read JS is harder
  • robots blocking, and bot detection (e.g. request rates, hidden link traps)
  • data poisoning. Put in books and links that nobody will want to have, that stall the download for bots that blindly collect everything.
  • mutation. Frequently change your templates, so that the scrapers may fail to find the desired contents.

Note that you can use Captchas very flexible.

For example: first book for each IP every day is non-captcha protected. But in order to access a second book, a captcha needs to be solved.

4
  • 2
    To be honest that doesn't solve my issue, but this answer is probably good for other people who'll come across this question on stackoverflow which is why I'm accepting it. I've come to accept that my problem can't really be solved. Jan 19, 2013 at 7:25
  • Have you considered the approach which starts displaying captchas when a certain threshold was reached? Jan 19, 2013 at 9:52
  • Yes, the users hate it, we'll probably just accept the fact that the data that we send to the free users is public domain Jan 19, 2013 at 9:53
  • I've added two more techniques. Requiring JavaScript (to decode links etc.) and mutation (e.g. a daily rotation of templates, so that scrapers may need to be adjusted daily) Update, I didn't see that you have a Restful API, so that won't work. Jan 19, 2013 at 10:13
1

Since you found that many of the items listed by Anony-Mousse dont solve your problem, I wanted to come in and suggest an alternative. Have you explored third party platforms that offer web scraping protection as a service? I'm going to list some of the solutions available on the market and try to lump them together. For full disclosure, I am one of the co-founders of Distil Networks, one of the companies that I am listing.

Web Scraping protection as a core competency:

  • Distil Networks
  • Sentor Assassin

Web Scraping protection as a feature in a larger product suite:

My opinion is that companies that try to solve the bot problem as a feature dont effectively do it well. Its just not their core competency and many loopholes exist

  • Akamai Kona
  • F5 ASM module to the BigIP loadbalancer
  • Imperva Web Application Firewall appliance
  • Incapsula, Imperva's cloud Web Application Firewall

It might also be helpful to talk about some of the pitfalls of the points mentioned:

  • captchas for registration and non-registered users captchas have been proven to be ineefective thanks to OCR software and captcha farms
  • rate limiting for IPs This could have a really high false positive rate as it lumps together users behind a shared IP. Also could miss a lot of bots if they simply rotate or annonomize the IP they use
  • require JavaScript Selenium, Phantom, and dozens of other scraping tools render javascript
4
  • +1 thanks, this might be useful for people. The problem is "use web scraping protection as a service" does not explain the technical solutions. What technologies does "Distil Networks" use in general (that I don't have to implement if I use such a service) that would really help me against bots? Oct 19, 2013 at 7:35
  • 1
    There are a couple steps that we use at Distil to identify and stop web scrapers. The first step is to validate that someone is using the browser they are reporting in their user agent. This is the easiest to do. It can start with making sure the end user is running javascript, but we go beyond that to make sure the headers are structured the right way for each browser, the multithreading matches, even going as far as making sure the TCP connection matches the type of operating system the browser reports to be under.
    – Rami
    Oct 21, 2013 at 2:38
  • 2
    The second step is to identify automation tools that run within browsers or headless browsers. This is harder to do and involves a lot of proprietary technology that I cant exactly disclose. Some things we do are inject hidden links, hidden css, and more to catch automation. The last step is behavioral analytics. We use machine learning to build a behavioral profile of your traffic pattern and find bot anomalies to your typical usage. Hopefully this was helpful. Our guys also blog a lot, so you can check out the blog for even more details: distilnetworks.com/blog
    – Rami
    Oct 21, 2013 at 2:42
  • Another third party solution that I can recommend from personal experience is named fireblade.
    – DJG
    Nov 21, 2013 at 7:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.