-1

I'm trying to pull data from this page using Scrapy: https://www.interpol.int/notice/search/woa/1192802

The spider will crawl multiple pages but I have excluded the pagination code here to keep things simple. The problem is that the number of table rows that I want to scrape on each page can change each time.

So I need a way of scraping all the table data from the page no matter how many table rows it has.

First, I extracted all the table rows on the page. Then, I created a blank dictionary. Next, I tried to loop through each row and put it's cell data into the dictionary.

But it does not work and it is returning a blank file.

Any idea what's wrong?

# -*- coding: utf-8 -*-
import scrapy


class Test1Spider(scrapy.Spider):
    name = 'test1'
    allowed_domains = ['interpol.int']
    start_urls = ['https://www.interpol.int/notice/search/woa/1192802']

    def parse(self, response):
        table_rows = response.xpath('//*[contains(@class,"col_gauche2_result_datasheet")]//tr').extract()
        data = {}
        for table_row in table_rows:
            data.update({response.xpath('//td[contains(@class, "col1")]/text()').extract(): response.css('//td[contains(@class, "col2")]/text()').extract()})
        yield data

2 Answers 2

2

What is this?

response.css('//td[contains(@class, "col2")]/text()').extract()

You are calling css() method but you are giving it a xpath

Anyways, here is the 100% working code, I have tested it.

table_rows = response.xpath('//*[contains(@class,"col_gauche2_result_datasheet")]//tr')
data = {}
for table_row in table_rows:
    data[table_row.xpath('td[@class="col1"]/text()').extract_first().strip()] = table_row.xpath('td[@class="col2 strong"]/text()').extract_first().strip()
yield data

EDIT:

To remove the characters like \t\n\r etc, use regex.

import re
your_string = re.sub('\\t|\\n|\\r', '', your_string) 
0
0

Try this.

I Hope it will help you.

# -*- coding: utf-8 -*-
import scrapy


class Test(scrapy.Spider):
    name = 'test1'
    allowed_domains = ['interpol.int']
    start_urls = ['https://www.interpol.int/notice/search/woa/1192802']

    def parse(self, response):
        table_rows = response.xpath('//*[contains(@class,"col_gauche2_result_datasheet")]//tr')

        for table_row in table_rows:
            current_row = table_row.xpath('.//td/text()').extract()
            print(current_row[0] + current_row[1].strip())

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.