8

I need to parse XML files of 40GB in size, and then normalize, and insert to a MySQL database. How much of the file I need to store in the database is not clear, neither do I know the XML structure.

Which parser should I use, and how would you go about doing this?

12
  • 1
    For large files, always use a pull parser like XMLReader; just try loading 40GB of file into memory for SimpleXML.
    – Mark Baker
    Mar 12, 2013 at 14:25
  • 1
    from the related posts: stackoverflow.com/questions/911663/… Mar 12, 2013 at 14:26
  • 40 Gb? Wow, I had trouble with a 700 Mb xml file on a mac os x machine. Mar 12, 2013 at 14:26
  • 1
    I would suggest you to write your own XML parser with minimum features (only the features you want). This way you have the most light-wieght XML parser possible and of course keep optimizing regularly
    – Girish
    Mar 12, 2013 at 14:27
  • 5
    @Girish - that's a crazy suggestion. Some of the best programmers on the planet have written highly optimized XML parsers, the chance of the average SO poster being able to do better is very small. Mar 12, 2013 at 18:34

2 Answers 2

13

In PHP, you can read in extreme large XML files with the XMLReaderDocs:

$reader = new XMLReader();
$reader->open($xmlfile);

Extreme large XML files should be stored in a compressed format on disk. At least this makes sense as XML files have a high compression ratio. For example gzipped like large.xml.gz.

PHP supports that quite well with XMLReader via the compression wrappersDocs:

$xmlfile = 'compress.zlib://path/to/large.xml.gz';

$reader = new XMLReader();
$reader->open($xmlfile);

The XMLReader allows you to operate on the current element "only". That means it's forward-only. If you need to keep parser state, you need to build it your own.

I often find it helpful to wrap the basic movements into a set of iterators that know how to operate on XMLReader like iterating through elements or child-elements only. You find this outlined in Parse XML with PHP and XMLReader.

See as well:

2

It would be nice to know what you actually intend to do with the XML. The way you parse it depends very much on the processing you need to carry out, as well as the size.

If this is a one-off task, then I've started in the past by discovering the XML structure before doing anything else. My DTDGenerator (see saxon.sf.net) was written for this purpose a long time ago and still does the job, there are other tools available now but I don't know whether they do streamed processing which is a prerequisite here.

You can write an application that processes the data using either a pull or push streamed parser (SAX or StAX). How easy this is depends on how much processing you have to do and how much state you have to maintain, which you haven't told us. Alternatively you could try streamed XSLT processing, which is available in Saxon-EE.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.