You can make it in one xpath-selector: //div/a/following-sibling::text()
for descriptions and just div ::text
for all the texts.
Check this example from scrapy shell
:
In [1]: txt = """<div>
...: <a href="#">Option 1</a>
...: Option 1 Description
...: <a href="#">Option 2</a>
...: Option 2 Description
...: <a href="#">Option 3</a>
...: Option 3 Description
...: </div>"""
In [2]: from scrapy import Selector
In [3]: sel = Selector(text=txt)
In [4]: sel.xpath("//div/a/following-sibling::text()").extract()
Out[4]:
[u'\n Option 1 Description\n ',
u'\n Option 2 Description\n ',
u'\n Option 3 Description\n']
In [5]: sel.css('div ::text').extract()
Out[5]:
[u'\n ',
u'Option 1',
u'\n Option 1 Description\n ',
u'Option 2',
u'\n Option 2 Description\n ',
u'Option 3',
u'\n Option 3 Description\n']