2

I have been evaluating the science of web scraping. The framework I use to do so is Python/Scrapy. I am sure there may be many more. My question is more around the basics. Suppose I have to scrape news content. So, I crawl a page and then write selectors to extract content, images, author, published date, sub description, comments etc. Writing this code is no big deal.

The question is how can I optimize this so it is scalable to large number of data sources. For instance, there may be thousands of news sites, each with its own html/page structure, so inevitably I need to write a scraping logic for EACH ONE OF THEM. Although possible, this would require a big team of resources working for a large duration of time to create and update these crawlers/scrapers.

Is there an easy way to do this? Can I somehow ease the process of creating a different scraper for each and every data source (website)?

How do sites like recordedfuture do it? Do they also have a big team working round the clock as they claim to extract data from 250000+ DISTINCT sources?

1 Answer 1

1

I'm not sure how RecordedFuture works, but it seems to me like a lot of the sites they scrape are their own clients, who are concerned about security threats.

I have not been able to write code general enough to parse many websites' data simultaneously. However, it is definitely possible to write code general enough to download web pages from many sites, provided that you know the final URLs or have general enough criteria for crawling each site (e.g., you will download every image).

I always download the HTML and then parse it later, so that I can iterate on my parsing and don't rely on the site staying live. Please let me know if this helps, and let me know more details about your use case so that I can better help you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.