I am working on using Hadoop Map Reduce to do research on the wikipedia data dumps (compressed in bz2 format). Since these dumps are so big (5 T), I can't decompress the xml data into HDFS and just use the StreamXmlRecordReader that hadoop provides. Hadoop does support uncompressing bz2 files, but it splits the pages arbitrarily and sends those to the mapper. Because this is xml, we need the splits to be a tags. Is there anyway to use the built in bz2 decompression and stream xml record reader provided by hadoop together?
-
Why do you need to split pages by tags?– svickCommented Jul 17, 2011 at 20:10
-
We want them split by <page> tags to be able to use a parser in python to get the data we need to analyze (we will be doing different types of analysis on the previous revisions and text of all pages).– Laurel OrrCommented Jul 17, 2011 at 20:24
-
Splitting by <page> tags will not be feasible as there are numerous pages which are over 100Gb long. See my full answer about the InputReader we just released.– DrDeeCommented Aug 6, 2011 at 11:15
2 Answers
The Wikimedia Foundation just released an InputReader for the Hadoop Streaming interface that is able to read the bz2 compressed full dump files and send it to your mappers. The unit being send to a mapper is not a whole page but two revisions (so you can actually run a diff on the two revisions). This is the initial release and I am sure there will be some bugs but please give it a spin and help us test it.
This InputReader requires Hadoop 0.21 as Hadoop 0.21 has streaming support for bz2 files. The source code is available at: https://github.com/whym/wikihadoop
Your problem is the same as described here. So my answer is the same too You should create your own variation on TextInputFormat. In there you make a new RecordReader that skips lines until it sees the start of a logical line.