0

I have a 456kb file which is being read from hdfs and its given as input to mapper function. Every line contain a integer for which I am downloading some files and storing them on local system. I have hadoop set up on two-node cluster and the split size is changed from the program to open 8-mappers :

    Configuration configuration = new Configuration();

    configuration.setLong("mapred.max.split.size", 60000L);
    configuration.setLong("mapred.min.split.size", 60000L);

8 mappers are created but same data is downloaded on both the servers, I think its happening because block size is still set to default 256mb and input file is processed twice. So my question is can we process a small size file with map reduce?

6
  • The framework doesn't stop you from processing small files. But, I didn't quite get what exactly you are trying to achieve.
    – Tariq
    Oct 3, 2013 at 21:34
  • I want this single file of 456kb to be processed by a number of mappers, instead of splitting the file the entire file is being processed twice on each server. So I am getting same out put on both servers which should not happen.
    – mumbai
    Oct 3, 2013 at 21:42
  • Are you using a custom InputFormat/RecordReader?
    – Mike Park
    Oct 3, 2013 at 23:18
  • @climbage No Input is just a txt file.
    – mumbai
    Oct 3, 2013 at 23:47
  • So what InputFormat are you using? TextInputFormat?
    – Mike Park
    Oct 4, 2013 at 14:15

1 Answer 1

1

If your download of files take time, you might have suffered from what's called speculative execution of Hadoop, which is by default enabled. It's just a guess though, since, you said you are getting same files downloaded more than once.

With speculative execution turn on the same input can be processed multiple times in parallel, to exploit differences in machine capabilities. As most of the tasks in a job are coming to a close, the Hadoop platform will schedule redundant copies of the remaining tasks across several nodes which do not have other work to perform.

You can disable speculative execution for the mappers and reducers by setting the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution JobConf options to false, respectively.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.