6

I am working on using Hadoop Map Reduce to do research on the wikipedia data dumps (compressed in bz2 format). Since these dumps are so big (5 T), I can't decompress the xml data into HDFS and just use the StreamXmlRecordReader that hadoop provides. Hadoop does support uncompressing bz2 files, but it splits the pages arbitrarily and sends those to the mapper. Because this is xml, we need the splits to be a tags. Is there anyway to use the built in bz2 decompression and stream xml record reader provided by hadoop together?

3
  • Why do you need to split pages by tags?
    – svick
    Commented Jul 17, 2011 at 20:10
  • We want them split by <page> tags to be able to use a parser in python to get the data we need to analyze (we will be doing different types of analysis on the previous revisions and text of all pages).
    – Laurel Orr
    Commented Jul 17, 2011 at 20:24
  • Splitting by <page> tags will not be feasible as there are numerous pages which are over 100Gb long. See my full answer about the InputReader we just released.
    – DrDee
    Commented Aug 6, 2011 at 11:15

2 Answers 2

7

The Wikimedia Foundation just released an InputReader for the Hadoop Streaming interface that is able to read the bz2 compressed full dump files and send it to your mappers. The unit being send to a mapper is not a whole page but two revisions (so you can actually run a diff on the two revisions). This is the initial release and I am sure there will be some bugs but please give it a spin and help us test it.

This InputReader requires Hadoop 0.21 as Hadoop 0.21 has streaming support for bz2 files. The source code is available at: https://github.com/whym/wikihadoop

0

Your problem is the same as described here. So my answer is the same too You should create your own variation on TextInputFormat. In there you make a new RecordReader that skips lines until it sees the start of a logical line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.