1

Assuming the following pattern:

<div>
  <a href="#">Option 1</a>
  Option 1 Description
  <a href="#">Option 2</a>
  Option 2 Description
  <a href="#">Option 3</a>
  Option 3 Description
  ...
</div>

I would like to obtain the text within each a tag and the following description text (which is not wrapped in tags).

What would be the best way to do so?

2 Answers 2

2

I have figured out a possible solution:

a_tags = response.css('div a')

for a in a_tags:
    command = a.xpath('text()').get()
    description = a.xpath('./following-sibling::text()[1]').get().strip()
0

You can make it in one xpath-selector: //div/a/following-sibling::text() for descriptions and just div ::text for all the texts.

Check this example from scrapy shell:

In [1]: txt = """<div>
   ...:   <a href="#">Option 1</a>
   ...:   Option 1 Description
   ...:   <a href="#">Option 2</a>
   ...:   Option 2 Description
   ...:   <a href="#">Option 3</a>
   ...:   Option 3 Description
   ...: </div>"""

In [2]: from scrapy import Selector

In [3]: sel = Selector(text=txt)

In [4]: sel.xpath("//div/a/following-sibling::text()").extract()
Out[4]: 
[u'\n  Option 1 Description\n  ',
 u'\n  Option 2 Description\n  ',
 u'\n  Option 3 Description\n']

In [5]: sel.css('div ::text').extract()
Out[5]: 
[u'\n  ',
 u'Option 1',
 u'\n  Option 1 Description\n  ',
 u'Option 2',
 u'\n  Option 2 Description\n  ',
 u'Option 3',
 u'\n  Option 3 Description\n']

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.