1

I am building a web application as college project (using Python), where I need to read content from websites. It could be any website on internet.

At first I thought of using Screen Scrapers like BeautifulSoup, lxml to read content(data written by authors) but I am unable to search content based upon one logic as each website is developed on different standards.

Thus I thought of using RSS/ Atom (using Universal Feed Parser) but I could only get content summary! But I want all the content, not just summary.

So, is there a way to have one logic by which we can read a website's content using lib's like BeautifulSoup, lxml etc?

Or I should use API's provided by the websites.

My job becomes easy if its a blogger's blog as I can use Google Data API but the trouble is, should I need to write code for every different API for the same job?

What is the best solution?

1

2 Answers 2

7

Using the website's public API, when it exists, is by far the best solution. That is quite why the API exists, it is the way that the website administrators say "use our content". Scraping may work one day and break the next, and it does not imply the website administrator's consent to have their content reused.

0

You could look into content extraction libraries - I've used Full Text RSS (php) and Boilerpipe (java).
Both have web service available so you can easily test if it meets your requirements. Also you can download and run them yourself and further modify its behavior on individual sites.

2
  • I am looking for a python lib
    – Surya
    Jun 17, 2012 at 19:03
  • I tried the Full Text RSS.. Its just showing the text but I need images/ slides.. everything.
    – Surya
    Jun 17, 2012 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.