-1

In my map reduce program, the mapper function will give two key value pair:

1) (person1, age)

2) (person2, age)

(I have kept 2 pairs only for simplicity it would be nice if u can explain for n nos of line)

Now I want to write a reducer which will compare age of both and give the answer who is older.

The thing I cannot understand is the output of mapper will be in different line in the file. And as reducer works on line by line bases over a file how will it compare them. Thanks in advance.

3 Answers 3

1

See if any of the following logic serves your purpose:

A.

  1. emit (age,person_name) from your map

  2. have only 1 reducer - your will get all ages, person pair in sorted manner. so simply emitting will give first one as youngest and last one as oldest.

    If you don't want to print all values, just have two references in the reducer task - youngest, oldest - set them in reduce method and emit whichever you want in the cleanup of reducer task

B.

  1. Have a mapper emitting (name,age) as you said
  2. in reducer task:

    a. Use setup() to create a treemap

    b. in reduce() add (age, person) in the treemap

    c. you map will be sorted by age which you can use in cleanup() to do something about it.

Essentially you can store all key,value in internal object(s) in reduce(), in cleanup() you will have access to all of these value and perform any logic you want in it.

7
  • That's Great!!!! I was looking for the functions, setup() and cleanup(), as u mentioned. Thank you Oct 31, 2015 at 9:59
  • The clean up method is called once per task(for mapper or reducer). So for reducer, this is still at each key i suppose
    – Ramzy
    Oct 31, 2015 at 13:32
  • @Ramzy, reducer task contains following methods in the order: 1. setup() 2. for each key,value reduce() 3. cleanup() so reduce() will have only one key and its values but you don't have to emit them, just store them in internal data structure (hashmap, list). In cleanup() you have all the keys and values you stored in your internal object, where you can perform your logic.
    – K246
    Oct 31, 2015 at 16:46
  • Ok, assumiing this is for option B, then reducer task would have handled part of all the keys, unless its a single reducer. So can we go ahead with this, to compare all keys from all mapper outputs?
    – Ramzy
    Oct 31, 2015 at 17:32
  • 1
    @Ramzy, Single reducer is still required to have all the data in the cleanup(). Also as Ravindra mentioned these are not efficient solutions with very large data as reducer memory can be bottleneck. I just wanted to point out how we can get rid of the limitation of knowing only one key, values limation in recue(). This approach can be useful in scenarios where mapper are processing very large amount of data but reducer input is not very large (so single reducer is ok).
    – K246
    Nov 1, 2015 at 1:01
1

I think your use case straight away fits into Secondary Sorting technique.

Secondary sorting is a technique, which has been introduced to sort "value" emitted by mapper. Primary sorting will be done by "key" emitted by mapper.

If you try to sort all values at reducer level, you may get out of memory. Secondary Sorting should be done at mapper level.

Have a look at this article

In above example, just replace "year" with "person" and "temperature" with "age"

Solution:

  1. Create Custom partitioner to send all values from a particular key to a single reducer

  2. Sorting should be done Key, Value combination emitted by mapper => Create a composite key with Key + Value have been used for sorting. Come up with a Comparator which sorts first by Key and then with Value.

0

In reducer method, all you would be getting is a key and list of values. So you can find min or max among a list of values for that key. However if you need to compare with other keys, then may be you should think of a single reducer and get all the records from mappers and handle that logic in your reducer class with help of reference variable rather than local variable and updating the reference variable with every min/max value for each key

6
  • You are right, if I have to compare values of same key it is easy, but in my case I want to compare value having different key. As u said I can use reference variable but issue in that is I cannot sort all values till I get all the values. So how would I know that this is the last call to reducer after this there is no data? Oct 30, 2015 at 13:57
  • When you say you want only one reducer, then all the data from all the mappers would be available and you can perform the comparision. Set number of reducer tasks to 1, and it should take care.
    – Ramzy
    Oct 30, 2015 at 14:05
  • There is one reducer only but mapper will give suppose n records having different keys. Now if I want to compare values associated with key 1 with the value associated with key 2 then how its that possible. Actually this is also possible, but for sorting all this n records how do I know which is the last record given by mapper.(Here by one record I mean a <key, values> pair forwarded by mapper) Oct 30, 2015 at 14:19
  • once all the mappers are done, since you have only one reducer, then the reducer method will be called for each key. So number of time the reducer method is called is equal to number of keys. So you can check for all values for that key and get minimum, and keep a reference of instance variable and updated as the reducer method is called.
    – Ramzy
    Oct 30, 2015 at 15:46
  • For minimum and maximum it is right that one reference variable will be enough. But if I need to do sorting on values having different key then? Oct 30, 2015 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.