2

I am new to Hadoop and MapReduce. I have some directory and files within this (each file 10 MB big and N could be 100. Files may be compressed or uncompressed) like: MyDir1/file1 MyDir1/file2 ... MyDir1/fileN

MyDir2/file1 MyDir2/file2 ... MyDir3/fileN

I want to design a MapReduce application where one mapper or reducer would process entire MyDir1 i.e. I dont want the MyDir1 to be split across multiple mappers. Similarly I want MyDir2 to be processed by other mapper/reducer completely without splitting.

Any idea on how to go about this? Do I need to write my own InputFormat and read the input files?

1
  • I actually have the same 2 requirements. I need the file not to be split because there is header information at the top of the file. I need a directory per mapper so that I can process the files in that directory in order as sorting the files (by date/time) is much more efficient than sorting individual rows.
    – MikeKulls
    Aug 29, 2014 at 4:55

1 Answer 1

6

Implement the FileInputFormat#isSplitable(). Then the input files are not split and are processed one per map. Note that the time to complete the job depends on the time to processes the largest input file, in spite of mappers executing in parallel. Also, this might not be efficient as there will be a lot of data shuffling across nodes.

import org.apache.hadoop.fs.*;
import org.apache.hadoop.mapred.TextInputFormat;
public class NonSplittableTextInputFormat extends TextInputFormat {
    @Override
    protected boolean isSplitable(FileSystem fs, Path file) {
        return false;
    }
}

The current API doen't allow a whole directory to be processed by a single mapper. You might have to write your own InputFormat. Or else create a list of directories to be processed and pass a single directory to each mapper to be processed, again this is not efficient because of data shuffling between nodes.

Coming back to reducers, they operate on the output KV pairs from the mappers and not the input files/directories.

3
  • If this class is defined within another class, as appears to be the standard with MR, then it will need to be marked as static.
    – MikeKulls
    Sep 2, 2014 at 0:22
  • @Praveen : can you tell me why there would be lot of data shuffling in this case? Nov 17, 2015 at 4:20
  • @vishnu : Because, in that particular approach, there is no guarantee that data is present in the node the map is running. So, the map process most likely to pull the data from another node
    – Bala
    Jun 29, 2016 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.