5

I'm wondering if it is possible to add a member object that can be used across multiple map() call. For example, a StringBuilder:

private StringBuilder builder;

public void map(...){
    ...

    builder.setLength(0);
    builder.append(a);
    builder.append(b);
    builder.append(c);
    d = builder.toString();

    ...
}

Obviously, if the mapper object is shared across multiple threads, the builder object above will not behave as expected due to concurrent access from more than one threads.

So my question is: Is it assured that each thread in hadoop will use one dedicated mapper object for itself? Or it is a configurable behavior?

Thanks

2 Answers 2

2

As long as you are not using the MultithreadedMapper class, but your own, there will be no problem. map() is called sequential and not in parallel.

It is common to use a StringBuilder or other data structures to buffer a few objects between the calls. But make sure you clone the objects from your input objects, there is only one object and it will be filled over and over again to prevent lots of GC.

So there is no need to synchronize or take care of race conditions.

2
  • That's what I want to know. Thanks.
    – JRaSH
    Apr 16, 2012 at 11:30
  • 1
    You only need to clone the objects if you need the entire Key/Value object, otherwise if you just need some of the contents that you add to the StringBuffer, you'll be fine. Don't forget to add the cleanup method (I forget the name to override) to flush/write to the Reducer the whatever you are buffering between Map calls, otherwise you'll end up with some data never getting sent on to the Reducers.
    – Drizzt321
    Apr 16, 2012 at 23:22
0

I don't think that's possible. The reason for that is that each mapper runs in its own JVM (they will be distributed on different machines), so there's no way you can share a variable or object across multiple mappers or reducers easily.

Now if all your mappers run on the same node, I believe there is a configuration for JVM reuse somewhere, but honestly I wouldn't bother with that, especially if all you need is a StringBuilder :)

I've seen this question once before, and it could be solved very easily by changing the design of the application. Maybe you can tell more about what you're trying to accomplish with this to see if this is really needed. If you really need it, you can still serialize your object, put it in HDFS, then read it with each mapper, deserialize it, but that seems backwards.

1
  • Hi, I don't mean to share a variable across multiple mappers. as far as i know, each mapper's map() method is called mutliple times, that's where i think i can save a few resources. So by putting a member object in the mapper instance, this objet can be resued across multiple map() call without init and de-init over and over. My concerns is: if map() is called concurrently, it might casue a problem for using memeber object.
    – JRaSH
    Apr 16, 2012 at 3:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.