0

I am trying to scrape data from a page with similar content(Shopping website) using HtmlAgility pack.

There is a button to load more items designed of tag. On click it loads more items on same page.

If it is designed using tag then I will get the next items using the href attribute URL in tag and also I will be loading new page for the new next items, So no problem.

But here no new URL and items loaded on same page.

So is there any way to get this functionality implemented? How to trigger that load more button to get more items?

0

1 Answer 1

0

HtmlAgilityPack is an html parser alone, it knows only to parse a static html document. what you want may be accomplished using selenium web driver.

Another possibility is - if the number of item load actions is so that you can complete the loading manually - do so and save the resulting html locally, and only afterwards use HtmlAgiliyPack to parse the static html you stored locally (instead of parsing the http response).

Share the link of the site you are talking about so I can add some code snippets to exemplify.

3
  • Ha, thanks for the suggestion. I am trying to do for this website like jabong.com or myntra.com or amazon.com or snapdeal.com or flipkart.com., all are of same type using to load items(jQuery or javascript). Few websites among these provide api's that we can use to get data. But I want to learn how apply scraping for javascript or jQuery or Ajax. Have a look and share the snippets if you code some.
    – Louis
    Jun 21, 2016 at 19:12
  • I was also wondering is there any packages that are used by python. Because I heard python is best way to scroll and scrape through pages. I mean google search does it.
    – Louis
    Jun 21, 2016 at 19:25
  • Yep, python is highly regarded in the scraping community. I think it's strength relies not much in the scraping itself, but in libraries that provide machine learning algorithms and stuff alike, which are ultimately essential in a scalable and wide range scraper. One famous library is Scrapy, other is BeautifulSoup. I am in the same boat as you, I ponder once in a while moving to Python.
    – Veverke
    Jun 22, 2016 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.