18

Goal: Extract text from a particular element (e.g. li), while ignoring the various mixed in tags, i.e. flatten the first-level child and simply return the concatenated text of each flattened child separately.

Example:

<div id="mw-content-text"><h2><span class="mw-headline" >CIA</span></h2>
    <ol>
    <li>Central <a href="/Intelligence_Agency.html">Intelligence Agency</a>.</li>
    <li>Culinary <a href="/Institute.html">Institute</a> of <a href="/America.html">America</a>.</li>
    </ol>

    </Div>  

desired text:

  • Central Intelligence Agency
  • Culinary Institute of America

Except that the anchor tags surrounding prevent a simple retrieval.

To return each li tag separately, we use the straightforward:

//div[contains(@id,"mw-content-text")]/ol/li

but that also includes surrounding anchor tags, etc. And

//div[contains(@id,"mw-content-text")]/ol/li/text()

returns only the text elements that are direct children of li, i.e. 'Central','.'...

It seemed logical then to look for text elements of self and descendants

//div[contains(@id,"mw-content-text")]/ol/li[descendant-or-self::text]

but that returns nothing at all!

Any suggestions? I'm using Python, so I'm open to using other modules for post-processing.

(I am using the Scrapy HtmlXPathSelector which seems XPath 1.0 compliant)

1

3 Answers 3

26

You were almost there. There is a small problem in:

//div[contains(@id,"mw-content-text")]/ol/li[descendant-or-self::text]

The corrected expression is:

//div[contains(@id,"mw-content-text")]/ol/li[descendant-or-self::text()]

However, there is a simpler expression that produces exactly the wanted concatenation of all text-nodes under the specified li:

string(//div[contains(@id,"mw-content-text")]/ol/li)
2
  • Is there a specific reason why contains is used instead of @id= or is it just because the OP asked the question with contains?
    – Kiril
    Feb 7, 2013 at 23:05
  • @Lirik, With this answer I help the OP to get his code doing what he wants -- I can't guess whether he wants to select a div with exactly that id attribute, or with an id attribute that contains a given string. It is likely he meant the former, but an answerer should avoid making guesses, whenever possible. Feb 7, 2013 at 23:30
5

I think the following would return the correct result:

//div[contains(@id,"mw-content-text")]/ol/li//text()

Note the double slash before text(). This means text nodes on any level below li must be returned.

2
  • This is a good idea, but it returns all text elements, without any context of which li they came from. Checking with Firefox's 'XPath Checker' I get: 1: Central 2: Intelligence Agency 3: . 4: Culinary 5: Institute 6: of 7: America 8: . There's no way to know which text came from which li...
    – ChaimKut
    May 16, 2012 at 12:34
  • If each line ends with a period (and there are no lines with periods in between (like Dr., Mr., etc.)), you can concatenate all text up to the period and just assume that each period == the end of an li. May 16, 2012 at 12:43
2

The string concatenation is tricky. Here's a quick solution using lxml:

>>> from lxml import etree
>>> doc = etree.HTML("""<div id="mw-content-text"><h2><span class="mw-headline" >CIA</span></h2>
...     <ol>
...     <li>Central <a href="/Intelligence_Agency.html">Intelligence Agency</a>.</li>
...     <li>Culinary <a href="/Institute.html">Institute</a> of <a href="/America.html">America</a>.</li>
...     </ol>
...
...     </Div>""")
>>> for element in doc.xpath('//div[@id="mw-content-text"]/ol/li'):
...   print "".join(element.xpath('descendant-or-self::text()'))
...
Central Intelligence Agency.
Culinary Institute of America.

Please note that // has potentially poor performance / unintended execution and should be avoided where possible, but difficult to do so with the example HTML fragment.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.