6

How do I get the first element that has an inner text (plain text, discarding other children) of 200 or more characters in length?

I'm trying to create an HTML parser like Embed.ly and I've set up a system of fallbacks where I first check for og:description, then I would search for this occurrence and only then for the description meta tag.

This is because most sites that even include meta description describe their site in that tag, instead of the contents of the current page.

Example:

<html>
    <body>
        <div>some characters
            <p>200 characters <span>some more stuff</span></p>
        </div>
    </body>
</html>

What selector could I use to get the 200 characters portion of that HTML fragment? I don't want the some more stuff either, I don't care what element it is (except for <script> or <style>), as long as it's the first plain text to contain at least 200 characters.

What should the XPath query look like?

3 Answers 3

8

Use:

(//*[not(self::script or self::style)]/text()[string-length() > 200])[1]

Note: In case the document is an XHTML document (and that means all elements are in the xhrml namespace), the above expression should be specified as:

(//*[not(self::x:script or self::x:style)]/text()[string-length() > 200])[1]

where the prefix "x:" must be bound to the XHTML namespace -- "http://www.w3.org/1999/xhtml" (or as many XPath APIs call this -- the namespace must be "Registered" with this prefix)

3
  • Since this is a crawling service I don't really have a clue about whether the document is XHTML or not
    – bevacqua
    Mar 6, 2012 at 3:22
  • @Nico: Then just try without the prefix -- I believe html agility pack doesn't require this. Mar 6, 2012 at 3:48
  • Yes, because any smart XPath engine will stop the evaluation after the first (and only required) node is found/selected. The expression you gave will traverse the whole tree, because it selects all such nodes. Mar 6, 2012 at 4:01
2

I meant something like this:

root.SelectNodes("html/body/.//*[(name() !='script') and (name()!='style')]/text()[string-length() > 200]")

Seems to work pretty well.

1
  • 1
    This XPath expression selects all such text node -- not just the first. This will take more space and time than necessary. In my answer I provide a more precise and efficient XPath expression. Mar 6, 2012 at 3:50
0

HTML is not XML. You should not use XML parsers to parse HTML period. They are two different things entirely, and your parser will choke out the first time you see html that's not well formed XML.

You should find an opensource HTML parser instead of rolling your own.

1
  • I'm using HtmlAgilityPack, I'm not trying to parse the HTML, I'm just trying to get the first portion of HTML with at least 200 characters in it.
    – bevacqua
    Mar 6, 2012 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.