39

Just wondering if anyone knows of a web-scraping library that takes advantage of Scala's succinct syntax. So far, I've found Chafe, but this seems poorly-documented and maintained. I'm wondering if anyone out there has done scraping with Scala and has advice. (I'm trying to integrate into an existing Scala framework rather than use a scraper written in, say, Python.)

5
  • The scraper libs that somewhat worked for me are httpunit and htmlunit. But they're Java libraries and not explicitly Scala. Commented Feb 7, 2013 at 12:28
  • 1
    When faced with a similar challenge I found this Scala HTML library useful. It's a bit old, but still did the trick for me. I wrapped it with an SBT build here. HTH Commented Feb 7, 2013 at 13:28
  • Just to inform that I have written a pimped Jsoup: github.com/filosganga/ssoup Commented May 26, 2013 at 10:25
  • 1
    I've written a DSL to build scraping Akka actor systems, it uses JSoup and spray-client under the hood. The DSL provided manipulates an immutable context (inspired by spray-routing), it can found here: github.com/bfil/scalescrape
    – BFil
    Commented Nov 23, 2014 at 16:11
  • 1
    I really like this article showing how to do scraping with akka: foat.me/articles/crawling-with-akka Commented May 20, 2017 at 11:12

3 Answers 3

14

First there is a plethora of HTML scraping libs in JVM all you need to do is pimp one of them (pimp my library pattern).

The four I have used are:

  • HtmlUnit - Will emulate the browser and even run Javascript
  • Jericho - Preserves formatting and ideal if you want to edit the scraped HTML
  • NekoHtml
  • JSoup -- does not work with Scala. Might work

I have used Selenium but never for scraping. Scala has a wrapper around selenium.

I would recommend pimping an existing Java library over some half baked Scala lib.

3
  • 2
    "half baked Scala" can be twice as good as "(over) baked java" though. Commented Jan 14, 2015 at 17:29
  • Well if its over baked yes. Nobody likes burnt cookies :) .... Then again you can't get salmonella from over baked cookies like you can from under baked cookies :) . Regardless I still think you should not play language bias particularly since Groovy, Scala, and Clojure are good at interacting with existing Java. Besides Java 8 is out so hopefully there will be less over baking.
    – Adam Gent
    Commented Jan 14, 2015 at 18:08
  • These days there is also scala-scraper library which is a wrapper around JSoup. It brings quite convenient syntax for extracting data from html pages. Commented Oct 28, 2015 at 8:22
8

I don't have a Scala-specific recommendation, but for the JVM in general I've had good success with:

  • JSoup You can CSS selectors to "scrape" the document. Really nice to work with.
  • Use Tagsoup to get your input HTML to XML, then use XML processors to "Scrape".

The Tagsoup route actually works quite well with Scala since Scala's built-in XML "dsl" is pretty concise (if you can forgive its perf issues and occasional API weirdness). Also, Tagsoup will handle nearly any garbage document you give it. It also has niceties like built-in understanding of many HTML entities that other SAXParsers will choke on as being undeclared.

tl;dr - JSoup + CSS selectors if possible, otherwise Tagsoup + scala XML. If slow is ok, tagsoup first, then jsoup the result.

5
  • FYI that last time I checked JSoup does not work with Scala: issues.scala-lang.org/browse/SI-3809
    – Adam Gent
    Commented Feb 7, 2013 at 19:58
  • @AdamGent Weird. I'm successfully using JSoup 1.7.1 in a project with scala 2.9.2. It's possible my use of it doesn't trigger the issue (?)
    – overthink
    Commented Feb 7, 2013 at 20:05
  • Probably. We found this out the hard way and I no longer have the code. Actually sadly it was one of the many reasons we stopped using Scala :(
    – Adam Gent
    Commented Feb 7, 2013 at 20:11
  • 3
    Looks like the JSoup author fixed the Scala issue a couple years back: github.com/jhy/jsoup/pull/54
    – overthink
    Commented Feb 7, 2013 at 20:13
  • Damn what a reminder of how long its been. If I remember correctly I don't even think Scala 2.9 was out when it happened.
    – Adam Gent
    Commented Feb 7, 2013 at 20:15
2

I'd recommend Goose: https://github.com/jiminoc/goose

It's not as general-use as you might need but if you are scraping article content from popular sites, it may work out of the box. It also provides a framework for you to work from if you want to extend their code to cover other sites.

Not the answer you're looking for? Browse other questions tagged or ask your own question.