2

I am new to crawl the webpage using Scrapy and unfortunately chose a dynamic one to start...

I've successfully crawled part (120 links), thanks to someone helping me here, but not links in target website

After doing some research, I know that crawling ajax web is nothing different from those simple ideas:

•open browser developer tools, network tab

•go to the target site

•click submit button and see what XHR request is going to the server

•simulate this XHR request in your spider

The last one sounds obscure to me though---How to simulate XHR request?

I've seen someone using 'headers' or 'formdata' and other parameters to simulate. Can't figure out what does that mean.

Here is part of my code:

class googleAppSpider(scrapy.Spider):
name = "googleApp"
allowed_domains = ['play.google.com']
start_urls = ['https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0']

def start_request(self,response):
    for i in range(0,10): 
        yield FormRequest(url="https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0", method="POST", formdata={'start':str(i+60),'num':'60','numChildren':'0','ipf':'1','xhr':'1','token':'m1VdlomIcpZYfkJT5dktVuqLw2k:1455483261011'}, callback=self.parse)

def parse(self,response):
    links = response.xpath("//a/@href").extract()
    crawledLinks = [ ]
    LinkPattern = re.compile("^/store/apps/details\?id=.")
    for link in links:
        if LinkPattern.match(link) and not link in crawledLinks:
            crawledLinks.append("http://play.google.com"+link+"#release")
    for link in crawledLinks:
            yield scrapy.Request(link, callback=self.parse_every_app)

def parse_every_app(self,response):

The start_request seems to not play any role here. If I delete them, the spider would still crawl the same amount of links.

I've worked on this problem for a week... Highly appreciate it if you could help me out...

1 Answer 1

0

Try this:

class googleAppSpider(Spider):
    name = "googleApp"
    allowed_domains = ['play.google.com']
    start_urls = ['https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0']

    def parse(self,response):
        for i in range(0,10): 
            yield FormRequest(url="https://play.google.com/store/apps/category/GAME/collection/topselling_new_free?authuser=0", method="POST", formdata={'start':str(i*60),'num':'60','numChildren':'0','ipf':'1','xhr':'1','token':'m1VdlomIcpZYfkJT5dktVuqLw2k:1455483261011'}, callback=self.data_parse)

    def data_parse(self,response):
        item = googleAppItem()
        map = {}
        links = response.xpath("//a/@href").re(r'/store/apps/details.*')
        for l in links:
            if l not in map:
                map[l] = True
                item['url'] = l
                yield item

Crawl the spider using scrapy crawl -o links.csv or scrapy crawl -o links.json you'll get all the links in a csv file or a json file. To increase the number of pages to crawl, change the range of for loop.

16
  • loop doesn't work...for everytime it loops,it passes to the next parse function. Feb 18, 2016 at 12:43
  • What does "doesn't works" means ? Aren't you getting all the links ?
    – Rahul
    Feb 18, 2016 at 12:45
  • still 120 links... not all links. interesting thing is if i delete that whole def function, i still get 120 Feb 18, 2016 at 13:02
  • If you will look closely, all the links after each request are completely different. You cannot get all the 540 links at once. This is what simulating xhr request means. Your browser sends a different xhr request whenever you scroll down. It doesn't get all the game links at once. This is exactly what the above code does. The loop is for each scroll down (each page).
    – Rahul
    Feb 18, 2016 at 13:06
  • Thanks... I ran the code exactly in cmd ( scrapy crawl ) and write the results into Json file. Then after I read the Json file into dataframe, I found out the length of that df is 120. It is how I find just crawl 120 links. Pls tell me what is wrong going on here... Feb 18, 2016 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.