I want to move on to the next page after scraping an eventsbrite page but it isn't working even after using Scrapy's Crawlspider.
here's the code to traverse the pages
allowed_domains = ["eventbrite.com"]
start_urls = ["https://www.eventbrite.com/d/nigeria--lagos/events/?crt=regular&end_date=01%2F31%2F2018&page=1&sort=best&start_date=12%2F01%2F2017",
]
def parse(self, response):
events = Selector(response).xpath('//div[@class="list-card-v2 l-mar-top-2 js-d-poster"]')
for event in events:
name = event.xpath('a/div[@class="list-card__body"]/div[@class="list-card__title"]/text()').extract()
venue = event.xpath('a/div[@class="list-card__body"]/div[@class="list-card__venue"]/text()').extract()
date = event.xpath('a/div[@class="list-card__body"]/time[@class="list-card__date"]/text()').extract()
event_type = event.xpath('a/div[@class="list-card__header"]/span/text()').extract()
category = event.xpath('div/div[@class="list-card__tags"]/a/text()').extract()
image= event.xpath('a/div[@class="list-card__header"]/div/img[@class="js-poster-image"]').extract()
image_url= event.xpath('a/div[@class="list-card__header"]/div/img[@class="js-poster-image"]/@src').extract()
name = ''.join(name).replace('\n', '').strip()
date = ''.join(date).replace('\n', '').strip()
venue = ''.join(venue).replace('\n', '').strip()
yield EventsItem(name=name, venue=venue, date=date,
event_type=event_type, category=category,
image_urls=image_url, images=image)
next_page = response.xpath('//a[@data-automation="next-page"]/@href').extract_first()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
here's an image of the the element. I dont know if it's because the href attribute is empty or a wrong xpath.
Any help is welcomed, thank you.
page=1
topage=2
and laterpage=3
, etc. – furas Dec 6 '17 at 16:34DevTool
in Chrome/Firefox then you will see that this portal use url eventbrite.com/directory/… to get next page asJSON
which Python can easily convert to dictionary and you will not have to load HTML. If you changepage=2
topage=3
in this url then you get third page as JSON. – furas Dec 6 '17 at 16:39