6

I want to make a code to extract the main news from a news website . News websites contain the main news , ads , reviews , copyright notice so i want to get only the main news like done in boilerpipe but i want to know how to do that .

So i want to have information about how is the process for doing this work .

Sudhanshu

4 Answers 4

8

the boilerpipe websites contains source code, quickstart instructions, links to the original scientific paper and to the corresponding conference presentation video:

http://code.google.com/p/boilerpipe/

This should give you a quite comprehensive set of information on how this works and how you can apply this in your scenario.

Best,

Christian

2

We try a lot of open source crawlers, like Readability, Beautiful Soup etc. for same, but after testing Diffbot API we decide use it for AppMarkt. It fast and extract news articles really well from various languages.

0

JSOUP provides an API for parsing HTML

0

I would give htmlcleaner a try.

HTMLCleaner is Java library used to safely parse and transform any HTML found on web to well-formed XML. It is designed to be small, fast, flexible and independant. HtmlCleaner may be used in java code, as command line tool or as Ant task. Result of parsing is lightweight document object model which can easily be transformed to standards like DOM or JDom, or serialized to XML output in various ways (compact, pretty printed and so on).

You can use XPath with htmlcleaner to get contents within xml/html tags.Here is a nice
example Xpath Example

Not the answer you're looking for? Browse other questions tagged or ask your own question.