2

I wrote a spider that scrapes ticket information from this website using code from another spider that works correctly (only needing to change the xpaths and the url. When I use the url in the terminal, I am able to access the response and extract different elements on the page from it. When I run the spider script, I get this error:

RLError: 2015-07-11 14:12:31 [boto] ERROR: Unable to read instance data, giving up

and the spider closes basically right after opening without scraping any elements. I've tested all of the xpaths and they are correct and I even copied the url directly from the webpage. Is there something different about this site that doesn't allow it to be scraped or is there a problem with the code? Btw, the site that I was able to scrape data from had a url that ended with .html so maybe it is that only those kinds of sites can be scraped or something. Any help would be appreciated on this, thanks.

import sys
import re
import json
from scrapy.crawler import CrawlerProcess
from scrapy import Request
from scrapy.contrib.spiders import CrawlSpider , Rule
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from scrapy.contrib.loader import ItemLoader
from scrapy.contrib.loader import XPathItemLoader
from scrapy.contrib.loader.processor import Join, MapCompose
from seat_geeks_scraper.items import ComparatorItem
from urlparse import urljoin

bandname = raw_input("Enter a bandname \n")
sg_url = "http://www.seatgeeks.com/" + bandname + "-tickets"
class MySpider2(CrawlSpider):
    handle_httpstatus_list = [416]
    name = 'comparator'
    allowed_domains = ["seatgeek.com"]
    start_urls = [sg_url]
    tickets_list_xpath = './/*[@itemtype="http://schema.org/Event"]'

    def parse_json2(self, response):
        loader = response.meta['loader']
        jsonresponse = json.loads(response.body_as_unicode())
        listings_info = jsonresponse.get('listings')
        price_list = [i.get('pf') for i in ticket_info]
        ticketPrice = price_list[0]
        loader.add_value('ticketPrice', ticketPrice)
        return loader.load_item()

    def parse_price2(self, response):
        loader = response.meta['loader']
        ticketsLink = loader.get_output_value("ticketsLink")
        json_id= ticketsLink.split('/')[6]
        json_url = "https://seatgeek.com/listings?client_id=MTY2MnwxMzgzMzIwMTU4&id=" + json_id + "&_wt=1&&_=1436364489501"
        yield scrapy.Request(json_url, meta={'loader': loader}, callback = self.parse_json, dont_filter = True) 

    def parse2(self, response):
        """

        # """
        selector = HtmlXPathSelector(response)
        # iterate over tickets
        for ticket in selector.select(self.tickets_list_xpath):
            loader = XPathItemLoader(ComparatorItem(), selector=ticket)
            # define loader
            loader.default_input_processor = MapCompose(unicode.strip)
            loader.default_output_processor = Join()
            # iterate over fields and add xpaths to the loader
            loader.add_xpath('eventName' , './/a[@class = "event-listing-title"]/span[@itemprop = "name"]/text()')
            loader.add_xpath('eventLocation' , './/a[@class = "event-listing-venue-link"]/span[@itemprop = "name"]/text()')
            loader.add_xpath('ticketsLink' , '//a[@class = "event-listing-button"]/@href')
            loader.add_xpath('eventDate' , '//div[@class = "event-listing-date"]/text()')
            loader.add_xpath('eventCity' , './/span[@itemprop  = "addressLocality"]/text()')
            loader.add_xpath('eventState' , './/span[@itemprop  = "addressRegion"]/text()')
            loader.add_xpath('eventCountry' , './/span[@itemprop  = "addressCountry"]/text()')
            loader.add_xpath('eventTime' , '//div[@class = "event-listing-time"]/text()')

            #ticketsURL = "concerts/" + bandname + "-tickets/" + bandname + "-" + loader.get_output_value("ticketsLink")
            tickets_url = "www.seatgeek.com/" + loader.get_output_value("ticketsLink")
            #ticketsURL = urljoin(response.url, ticketsURL)
            yield scrapy.Request(tickets_url, meta={'loader': loader}, callback = self.parse_price2, dont_filter = True)

enter image description here

enter image description here

9
  • That boto error relate to AWS services, are you sure it's about the scraping? By the way, allowed_domains contains an extra / at the end, you should remove it. What does scrapy crawl comparator2 stats at the end show? Jul 12, 2015 at 8:41
  • @paultrmbrth Sorry for the late reply! I just updated the post with screenshots of the terminal after running the spider. I think that it has something to do with the extension of the url because I was just able to scrape another website with .html extension, but this one isn't working. And I took out the slash in the allowed domains. Thanks Jul 17, 2015 at 19:07
  • Your site (seatgeek.com) is not a part of your allowed domains (www.seatgeek.com). Remove the 'www' prefix.
    – lufte
    Jul 17, 2015 at 21:33
  • @JavierAyres It didn't work. I just updated the code in the post with what you said Jul 17, 2015 at 21:40
  • I have the same issue. I'll let you know if I find a solution.
    – robert
    Jul 18, 2015 at 0:15

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.