I am trying to scrape data from two websites. I want to build a generic web parser to scrape those website. I need a generic parser because i might need to parse additional websites in the future. I want to parse the websites using java and store specific data in MySQL. What are the steps I should take? Can anyone guide me in this?
1 Answer
I've done this same thing for scraping the web using java and building a parser is an absolute nightmare, there are so many more things to consider that you realize at the start.
Use jSoup. It is sooo easy, and leaves you time to enjoy the finer things in life.
-
1Yes, that's what I plan to use. I think using Jsoup and regular expressions should make it a bit easy. Thanks!– NeminDec 6, 2013 at 21:37
-
1No worries! I really like too how jsoup has a "sandbox" so to speak where you can test stuff out - try.jsoup.org, and the main page for learning how to use it is their "cookbook" (took me a while to find it, but it's a gem)– Josh TDec 6, 2013 at 21:39
-