8

i am new to hadoop and i'm working with large number of small files in wordcount example. it takes a lot of map tasks and results in slowing my execution.

how can i reduce the number of map tasks??

if the best solution to my problem is catting small files to a larger file, how can i cat them?

2 Answers 2

7

If you're using something like TextInputFormat, the problem is that each file has at least 1 split, so the upper bound of the number of maps is the number of files, which in your case where you have many very small files you will end up with many mappers processing each very little data.

To remedy to that, you should use CombineFileInputFormat which will pack multiple files into the same split (I think up to the block size limit), so with that format the number of mappers will be independent of the number of files, it will simply depend on the amount of data.

You will have to create your own input format by extending from CombineFileInputFormt, you can find an implementation here. Once you have your InputFormat defined, let's called it like in the link CombinedInputFormat, you can tell your job to use it by doing:

job.setInputFormatClass(CombinedInputFormat.class);
1
  • if i wanted to maintain the id of small files, what do i do? Is there any way?
    – csperson
    Feb 5, 2013 at 11:37
3

Cloudera posted a blog on small files problem sometime back. It's an old entry, but the suggested method still applies.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.