I get a url that cannot be used to fetch data from next page, so created a base_url = 'http://www.marinetraffic.com'
variable and passed it scrapy request. port_homepage_url = base_url + port_homepage_url
. It works fine, when i yeild the result like this. yield {'a': port_homepage_url, 'b':item['port_name']}
I get this result i wanted.
however if place it in scrapy request yield scrapy.Request(port_homepage_url, callback=self.parse, meta={'item': item})
i get error
port_homepage_url = base_url + port_homepage_url
TypeError: cannot concatenate 'str' and 'NoneType' objects
here is code
class GetVessel(scrapy.Spider):
name = "getvessel"
allowed_domains = ["marinetraffic.com"]
start_urls = [
'http://www.marinetraffic.com/en/ais/index/ports/all/flag:AE',
]
def parse(self, response):
item = VesseltrackerItem()
base_url = 'http://www.marinetraffic.com'
for ports in response.xpath('//table/tr[position()>1]'):
item['port_name'] = ports.xpath('td[2]/a/text()').extract_first()
port_homepage_url = ports.xpath('td[7]/a/@href').extract_first()
port_homepage_url = base_url + port_homepage_url
yield scrapy.Request(port_homepage_url, callback=self.parse, meta={'item': item})
ports.xpath('td[7]/a/@href').extract_first()
is returning None, that's for sure. Can't tell why, though – lucasnadalutti Sep 30 '16 at 15:13