What are some Ruby-based tools that can help me achive something similar to what Scrapy does for python? http://doc.scrapy.org/intro/overview.html
-
2You might want to just give scrapy a shot. It seems to have excellent documentation and a very friendly API. If you are familiar with ruby picking up python probably won't take you more than a day or two. They are extremely similar languages.– zeekayMay 6, 2011 at 16:15
-
@zeekay, do you mean scrappy?– ZabbaMay 6, 2011 at 16:26
-
1No, he asked about a ruby alternative to scrapy, which is a python application framework for crawling web sites and extracting structured data. I was just pointing out that learning python is easy, and he might as well just use it (assuming no satisfactory alternative is found).– zeekayMay 6, 2011 at 16:36
-
Ruby Toolbox for HTML parsing: ruby-toolbox.com/categories/html_parsing.html . HTTP clients: ruby-toolbox.com/categories/http_clients.html– Andrew GrimmMay 8, 2011 at 4:26
2 Answers
There's Mechanize which is built upon Nokogiri.
There's Nokigiri which is based on XPath.
Hpricot is another tool.
There's Scrapi which is based on CSS selectors to extract information, but performs slower than Nokogiri based on my testing.
There's scRUBYt.
I'm sure there are others, but these are the ones that I came across.
If you don't find a single tool that solves your problems, checkout web spidering libraries like Anemone and combine it with one of the low-level scraping frameworks listed above.
Or just go ahead and learn Python. It'll expand your karma in the programming world.
-
4As far as I can tell (I don't know Ruby), those modules are just HTML/XML parsers, not scraping frameworks like Scrapy.– AcornMay 6, 2011 at 17:20
-
1@Acorn - these are all scraping frameworks. What they don't have is a crawler or a spider. Scraping is orthogonal to crawling.– AnuragMay 6, 2011 at 17:42
-
6But.. crawling is only one aspect to Scrapy. It has many features that apply to any kind of scraping, such as Items and pipelines for storing results. The tools you linked to seem comparable to lxml or BeautifulSoup rather than Scrapy.– AcornMay 6, 2011 at 17:45
-
@Acorn - that's true, Nokogiri, Hpricot, and Scrapi are for extracting data from a single page. scRUBYt has the option of following links but their documentation is not very good. There is another tool that I didin't list - Mechanize which is built upon Nokigiri and seems to be a better tool for the job.– AnuragMay 6, 2011 at 18:27
-
1@Acorn, @Anurag - I'm looking for something more similar to Scrapy, that makes it easy to spider as well. Extracting from single pages with nogogiri is pretty easy, but traversing pages to fetch the links of individual postings is a pain. But I guess I could just go learn Python too :-)– AvishaiMay 8, 2011 at 4:41