0

I am running this code using oozie workflow and getting type mismatch error:

 public static class mapClass extends Mapper<Object, Text, LongWritable, LongWritable> {
        public void map(Object, Text, Context..)
..
         context.write(<LongWritable type> , <LongWritable type> )
}

public static class reduceClass extends Reducer<LongWritable, LongWritable,LongWritable, LongWritable> {
        public void reduce(LongWritable, LongWritable, context)
..
    context.write(<LongWritable type>, <LongWritable type>)
{

}

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.LongWritable, recieved org.apache.hadoop.io.Text

I'm using new-api in my workflow. The same code works fine without using oozie.

Any help would be appreciated. Thanks.

-----code sample ---

package org.apache.hadoop;
import java.io.IOException;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;


public class MapperLong extends Mapper<LongWritable, Text, LongWritable, LongWritable> {
    public final static int COL_ZERO = 0;
    public final static int COL_ONE = 1;
    public final static int COL_TWO = 2;
    public final static int COL_THREE = 3;

@Override
    public void map(LongWritable offset, Text line, Context context)
throws IOException, InterruptedException {
        String[] parts = (line.toString()).split(" ");

        LongWritable one = new LongWritable(Integer.parseInt(parts[COL_ONE]));
        LongWritable two = new LongWritable(Integer.parseInt(parts[COL_TWO]));
        context.write(one, two);

    }
}




package org.apache.hadoop;

import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.mapreduce.Reducer;


public class ReducerLong extends Reducer<LongWritable, LongWritable, LongWritable, LongWritable> {
@Override
    public void reduce(LongWritable colOneKey, Iterable<LongWritable> values,
        Context context) throws IOException, InterruptedException{
        Set<Integer> colTwo = new HashSet<Integer>();

        for (LongWritable val : values) {
            colTwo.add(Integer.valueOf((int)val.get()));
        }
        context.write(colOneKey, new LongWritable(colTwo.size()));
    }
}
}

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.LongWritable, recieved org.apache.hadoop.io.Text at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264)

Input :
34 342 1 1
45 23 0 1
..
..

Note: I changed Object type to LongWritable which did not make any difference. The above exception is thrown while using the following property in workflow.xml. Without the following property, code executes producing output same as input prefixed with offset!

        <property>
            <name>mapred.output.key.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>
        <property>
            <name>mapred.output.value.class</name>
            <value>org.apache.hadoop.io.LongWritable</value>
        </property>
4
  • The posted code look good. Could you post complete code? Dec 24, 2011 at 1:30
  • 1
    In general, don't forget that Java generics are not enforced at Runtime, so anybody can put any object in the map as they wish. So if something put in a Hadoop Text instance instead of a Hadoop LongWritable, you would get this message. More code, full stacktrace of exception would be helpful. Dec 24, 2011 at 2:38
  • Agreed with owlstead. I'd feel better if the Mapper's first template item is not Object. Dec 24, 2011 at 3:00
  • I have added the code for your reference.
    – hadoopQ
    Dec 27, 2011 at 17:31

2 Answers 2

2

Ok, I figured out. The problem was in the oozie workflow I defined.

I had 
<name>mapreduce.mapper.class</name>
and 
<name>mapreduce.reducer.class</name>

instead of
<name>mapreduce.map.class</name>
and 
<name>mapreduce.reduce.class</name>

[refer https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases MR Api ] For some reason that could not catch my eyes :-( since I was using the modified version of non-api workflow!

Thank you all for your time.

1

Most likely you are using the reducer as a combiner which means it runs in the context of the map. see a similar question here Wrong key class: Text is not IntWritable

1
  • 2
    The reducer is LongWritable, LongWritable -> LongWritable, LongWritable, and the mapper is outputting LongWritable, LongWritable so it should be fine... Dec 24, 2011 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.