1

I am trying scrapy to crawl a ajax website http://play.google.com/store/apps/category/GAME/collection/topselling_new_free

I want to get all the links directing to each game.

I inspect the element of the page. And it looks like this: how the page looks like so I want to extract all links with the pattern /store/apps/details?id=

but when I ran commands in the shell, it returns nothing: shell command

I've also tried //a/@href. didn't work out either but Don't know what is wrong going on....

  • Now I can crawl first 120 links with starturl modified and 'formdata' added as someone told me but no more links after that.

Can someone help me with this?

2
  • I am not familiar with scrapy, but have you tried Beautiful Soup? It is a popular screen sraping libray I had success with in the past.
    – Igor
    Feb 10, 2016 at 0:59
  • see view source, try to find those links, if you cant, those are written dynamically by javascript. you need something like phantomjs.
    – YOU
    Feb 10, 2016 at 2:16

1 Answer 1

1

It's actually an ajax-post-request which populates the data on that page. In scrapy shell, you won't get this, instead of inspect element check the network tab there you will find the request.

Make post request to https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0 url with formdata={'start':'0','num':'60','numChildren':'0','ipf':'1','xhr':'1'}

Increment start by 60 on each request to get the paginated result.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.