I am trying to scrape items starting with A from this page using following code.
import scrapy
from scrapy.selector import Selector
from ..items import RozeepkItem
class JobcatsSpider(scrapy.Spider):
name = 'jobcats'
allowed_domains = ['www.rozee.pk']
start_urls = ['https://www.rozee.pk/jobs-by-industry']
def parse(self, response):
items = RozeepkItem()
for job_cat in Selector(response).xpath("//div[@class = 'boxb job-dtl sitemap']"):
category_title = job_cat.xpath(".//div[@id = 'A-block']/div[@class = 'row']/ul/li/a/@title").get()
url = job_cat.xpath(".//div[@id = 'A-block']/div[@class = 'row']/ul/li/a/@href").get()
items['job_category'] = category_title
items['url_str'] = url
yield items
following is items.py
import scrapy
class RozeepkItem(scrapy.Item):
job_category = scrapy.Field()
url_str = scrapy.Field()
it gives output
2020-06-27 06:55:00 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.rozee.pk/jobs-by-industry>
{'job_category': 'Accounting Jobs in Pakistan',
'url_str': '//www.rozee.pk/search/accounting-jobs-in-pakistan'}
2020-06-27 06:55:00 [scrapy.core.engine] INFO: Closing spider (finished)
2020-06-27 06:55:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 232,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 16977,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'elapsed_time_seconds': 2.227556,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 6, 27, 1, 55, 0, 379725),
'item_scraped_count': 1,
'log_count/DEBUG': 2,
'log_count/INFO': 10,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2020, 6, 27, 1, 54, 58, 152169)}
2020-06-27 06:55:00 [scrapy.core.engine] INFO: Spider closed (finished)
As it can be seen I am only getting one item and its corresponding link, on the other hand if i try this xpath in browser I get all the information as it can be seen in following screenshot.
Can some help me that where I am making mistake? Thank you