32

I'm into some web scraping with Node.js. I'd like to use XPath as I can generate it semi-automatically with several sorts of GUI. The problem is that I cannot find a way to do this effectively.

  1. jsdom is extremely slow. It's parsing 500KiB file in a minute or so with full CPU load and a heavy memory footprint.
  2. Popular libraries for HTML parsing (e.g. cheerio) neither support XPath, nor expose W3C-compliant DOM.
  3. Effective HTML parsing is, obviously, implemented in WebKit, so using phantom or casper would be an option, but those require to be running in a special way, not just node <script>. I cannot rely on the risk implied by this change. For example, it's much more difficult to find how to run node-inspector with phantom.
  4. Spooky is an option, but it's buggy enough, so that it didn't run at all on my machine.

What's the right way to parse an HTML page with XPath then?

2

5 Answers 5

49
+50

You can do so in several steps.

  1. Parse HTML with parse5. The bad part is that the result is not DOM. Though it's fast enough and W3C-compiant.
  2. Serialize it to XHTML with xmlserializer that accepts DOM-like structures of parse5 as input.
  3. Parse that XHTML again with xmldom. Now you finally have that DOM.
  4. The xpath library builds upon xmldom, allowing you to run XPath queries. Be aware that XHTML has its own namespace, and queries like //a won't work.

Finally you get something like this.

const fs = require('mz/fs');
const xpath = require('xpath');
const parse5 = require('parse5');
const xmlser = require('xmlserializer');
const dom = require('xmldom').DOMParser;

(async () => {
    const html = await fs.readFile('./test.htm');
    const document = parse5.parse(html.toString());
    const xhtml = xmlser.serializeToString(document);
    const doc = new dom().parseFromString(xhtml);
    const select = xpath.useNamespaces({"x": "http://www.w3.org/1999/xhtml"});
    const nodes = select("//x:a/@href", doc);
    console.log(nodes);
})();

Note that you have to prepend every single HTML element of a query with the x: prefix, for example to match an a inside a div you would need:

//x:div/x:a
6
  • Thank you, works perfectly. Except that I needed to replace var document = parser.parse(html.toString()); by var document = parse5.parse(html.toString()); and get rid of the line var parser = new parse5.Parser(); (using parse5 version 2.0.2)
    – qqilihq
    Jan 4, 2016 at 22:31
  • 1
    You are loading everything in memory (the entire DOM)... is there a more memory efficient way to do this?
    – Fabiosoft
    Apr 16, 2019 at 15:20
  • I wondering if it is possible to create a custom parse5 treeAdapter that would avoid the serializeToString/parseFromString step ? (see github.com/inikulin/parse5/blob/master/packages/parse5/docs/…) Mar 5, 2020 at 13:30
  • 1
    @Fabiosoft Unfortunately XPath queries do require DOM. There were implementations of subset of XPath that worked over a SAX parser for PHP, but (I almost hope that) there is no such thing on npm. May 14, 2020 at 19:20
  • @FranckFreiburger If I were to do any web crawlers today, I'd just use CSS selectors. They lack features like walking back to some parent, but you won't need anything beyond a call to parse5. XML and tooling around it (like XPath or Java) fell out of mainstream some time back in 2014. May 14, 2020 at 19:26
16

Libxmljs is currently the fastest implementation (something like a benchmark) since it's only bindings to the LibXML C-library which supports XPath 1.0 queries:

var libxmljs = require("libxmljs");
var xmlDoc = libxmljs.parseXml(xml);
// xpath queries
var gchild = xmlDoc.get('//grandchild');

However, you need to sanitize your HTML first and convert it to proper XML. For that you could either use the HTMLTidy command line utility (tidy -q -asxml input.html), or if you want it to keep node-only, something like xmlserializer should do the trick.

2

I have just started using npm install htmlstrip-native which uses a native implementation to parse and extract the relevant html parts. It is claiming to be 50 times faster than the pure js implementation (I have not verified that claim).

Depending on your needs you can use html-strip directly, or lift the code and bindings to make you own C++ used internally in htmlstrip-native

If you want to use xpath, then use the wrapper already avaialble here; https://www.npmjs.org/package/xpath

5
  • 0. Your link is broken. 1. That library is parsing entities, and that's quite obvious from its name. 2. XPath is not even mentioned in your answer. Sep 21, 2014 at 20:24
  • Fixed the broken link; added link to the xpath implementation, any reason you didn't find/use that yourself?
    – Soren
    Sep 21, 2014 at 20:32
  • That xpath library has to be run over some kind of DOM. The only solution that parses HTML is jsdom, which is slow as hell. It's the first item in the list up there. Did you read the question? Sep 21, 2014 at 20:57
  • If you were to read the npm xpath documentation you would see that he suggest using the xmldom .
    – Soren
    Sep 21, 2014 at 22:03
  • And how xmldom is supposed to parse HTML? Sep 22, 2014 at 6:03
2

I think Osmosis is what you're looking for.

  • Uses native libxml C bindings
  • Supports CSS 3.0 and XPath 1.0 selector hybrids
  • Sizzle selectors, Slick selectors, and more
  • No large dependencies like jQuery, cheerio, or jsdom
  • HTML parser features

    • Fast parsing
    • Very fast searching
    • Small memory footprint
  • HTML DOM features

    • Load and search ajax content
    • DOM interaction and events
    • Execute embedded and remote scripts
    • Execute code in the DOM

Here's an example:

osmosis.get(url)
    .find('//div[@class]/ul[2]/li')
    .then(function () {
        count++;
    })
    .done(function () {
        assert.ok(count == 2);
        assert.done();
    });
0
1

With just one line, you can do it with xpath-html:

const xpath = require("xpath-html");

const node = xpath.fromPageSource(html).findElement("//*[text()='Made with love by']");
2
  • 4
    It's good that you've made a library that incapsulates an answer by @pda. If a better approach shows up, it's possible to update only one library. On the other hand it's a little bit shady that you don't mention this is your library, and that this library is basically another answer from this thread. May 14, 2020 at 19:10
  • Worth noting that this has one/some serious bugs right now: github.com/hieuvp/xpath-html/issues/10#issuecomment-752248148 Dec 30, 2020 at 10:20

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.