9

I'm wondering which XML parser people thing would be best in my situation for Perl. I've done a lot of reading and have tried the XML::LibXML and XML::SAX. The first used up too much memory and the second didn't appear to be that quick to me (even after switching off the pure perl parser).

My needs are fairly specific. I am receiving a largish response of up to 50MB via the Net::SSH library. I would like to pass this data to an XML library as I receive it so as to keep the minimum amount of data in memory. I need to then look for data in certain tags and do whatever with it, in some cases sum a bunch of values, in other cases just extract values and write them to files or whatever. So I need an XML parser that can work serially, works quick and uses the minimum of memory. The data I get is in chunks of up to 1024 bytes so I would like to be able to just do something like $myparser->sendData($mynewData) and then have functions called when a new tag is opened or closed similar to what XML::SAX does.

I don't necessarily need XPath or XSLT.

3 Answers 3

7

I would recommend using XML::Twig.

This module is very convenient to use, and also it can read data serially without using much memory.

Probably one of the most distinctive features of XML::Twig is that it permits to parse XML in so-called hybrid model: you can parse whole document (needs whole document and a lot of memory), you can use callbacks to parse small chunks (allows streaming, small memory consumption), or you can use any combination of these.

This combined model turns out to be most convenient feature - load small leaf from the stream, and you can access all its small branches effectively for free.

3
  • Thanks mvp and @IlmariKaronen I will try these and mark the green tick when I've worked out which one works best. It could take me a while :-)
    – MikeKulls
    Jan 3, 2013 at 20:55
  • I've tested several xml parsers now (as well as just using a regex) and the results i get for a 50MB file are regex less than 1 second, XML::Parser 21 sec, XML::SAX 22 seconds, twig 70 seconds. In all cases I just looked for the node named "interface" and counted them. Memory usage was minimal for all of them. It was interesting that all the parsers were significantly slower than regex. I'm leaning towards XML::Parser because it is already installed on the server, was simple to use and is fast. But maybe I am doing something wrong with twig?
    – MikeKulls
    Jan 3, 2013 at 23:34
  • If you were just counting nodes, you should have used XML::Twig ignore method or equivalents like setIgnoreEltsHandler - it will speed it up considerably, as twig will not need to be built. Also, you should call purge method in your callback to free memory allocated to twig - it will make it faster
    – mvp
    Jan 4, 2013 at 10:19
4

You could also go with plain old XML::Parser, which does pretty much just what you ask for:

"This module provides ways to parse XML documents. It is built on top of XML::Parser::Expat, which is a lower level interface to James Clark's expat library. Each call to one of the parsing methods creates a new instance of XML::Parser::Expat which is then used to parse the document. Expat options may be provided when the XML::Parser object is created. These options are then passed on to the Expat object on each parse call. They can also be given as extra arguments to the parse methods, in which case they override options given at XML::Parser creation time."

"Expat is an event based parser. As the parser recognizes parts of the document (say the start or end tag for an XML element), then any handlers registered for that type of an event are called with suitable parameters."

I've used it for parsing Wikipedia XML dumps, which are several GB in size even after compression, and found it to work very well for that. Compared to that, a 50 MB file should be a piece of cake.

4
  • Thanks Ilmari. How quick did you find it? I found that perl can load a 50MB file and search all the way through it in less than 1 second where as the XML parsers all seem to take about 20 seconds. Does that sound correct?
    – MikeKulls
    Jan 4, 2013 at 1:21
  • I just tried running one of my old scripts, and it's doing about 10 MB per second on the 1.7 GB latest Finnish Wikipedia dump on a 2 GHz Intel Xeon server. Jan 4, 2013 at 1:57
  • The XML parsers are slower than a pure regex because they account for all the various ways XML can be written, and that takes some time. Unless you have complete control over how the XML is written, though, that extra time will be the difference between doing it fast and doing it right (a simple regex can be sensitive to things like order of attributes in tags and use of ignorable whitespace, and can match things in the wrong place, e.g. in comments).
    – ebohlman
    Jan 7, 2013 at 2:58
  • @ebohlman I'm definately NOT interesting in using regex and think it is the biggest hack ever, but someone here at work is. I'm trying to find a solution that is comparable in performance so that we can make code that doesn't fall over with slight differences in the XML. I can see at least 10 ways in which the regex would fail. The 20 seconds is not really an issue because it takes up to 3 minutes for the response to come back and we can process the response as it goes.
    – MikeKulls
    Jan 7, 2013 at 23:55
3

XML::LibXML also comes with two stream parsers which should be faster than pure Perl solutions:

XML::LibXML::SAX

A SAX parser compatible with XML::SAX.

XML::LibXML::Reader

An interface to libxml2's pull parser. It provides a simpler interface than SAX and is a bit faster. To my knowledge, XML::LibXML::Reader is the fastest Perl module to parse XML files without loading them into memory completely.

1
  • Hi Nick, do you know if you can pass data in chunks to XML::LibXML::SAX? I could only see a method to parse a file serially but I am receiving the data from the perl ssh library. Something like the XML::Parser's parse_more function.
    – MikeKulls
    Jan 29, 2013 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.