0

The output folder has part-00000 file with no content!

Here is the command trace where I see no exception,

[cloudera@localhost ~]$ hadoop jar testmr.jar TestMR /tmp/example.csv /user/cloudera/output
14/02/06 11:45:24 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/06 11:45:24 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/02/06 11:45:24 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/02/06 11:45:25 INFO mapred.FileInputFormat: Total input paths to process : 1
14/02/06 11:45:25 INFO mapred.JobClient: Running job: job_local1238439569_0001
14/02/06 11:45:25 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/06 11:45:25 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
14/02/06 11:45:25 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/06 11:45:25 INFO mapred.LocalJobRunner: Starting task: attempt_local1238439569_0001_m_000000_0
14/02/06 11:45:26 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
14/02/06 11:45:26 INFO util.ProcessTree: setsid exited with exit code 0
14/02/06 11:45:26 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@44aea710
14/02/06 11:45:26 INFO mapred.MapTask: Processing split: hdfs://localhost.localdomain:8020/tmp/example.csv:0+2963382
14/02/06 11:45:26 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group name and  BYTES_READ as counter name instead
14/02/06 11:45:26 INFO mapred.MapTask: numReduceTasks: 1
14/02/06 11:45:26 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/06 11:45:26 INFO mapred.MapTask: io.sort.mb = 50
14/02/06 11:45:26 INFO mapred.MapTask: data buffer = 39845888/49807360
14/02/06 11:45:26 INFO mapred.MapTask: record buffer = 131072/163840
14/02/06 11:45:26 INFO mapred.JobClient:  map 0% reduce 0%
14/02/06 11:45:28 INFO mapred.MapTask: Starting flush of map output
14/02/06 11:45:28 INFO compress.CodecPool: Got brand-new compressor [.snappy]
14/02/06 11:45:28 INFO mapred.Task: Task:attempt_local1238439569_0001_m_000000_0 is done. And is in the process of commiting
14/02/06 11:45:28 INFO mapred.LocalJobRunner: hdfs://localhost.localdomain:8020/tmp/example.csv:0+2963382
14/02/06 11:45:28 INFO mapred.Task: Task 'attempt_local1238439569_0001_m_000000_0' done.
14/02/06 11:45:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local1238439569_0001_m_000000_0
14/02/06 11:45:28 INFO mapred.LocalJobRunner: Map task executor complete.
14/02/06 11:45:28 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
14/02/06 11:45:28 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1d382926
14/02/06 11:45:28 INFO mapred.LocalJobRunner: 
14/02/06 11:45:28 INFO mapred.Merger: Merging 1 sorted segments
14/02/06 11:45:28 INFO compress.CodecPool: Got brand-new decompressor [.snappy]
14/02/06 11:45:28 INFO mapred.Merger: Down to the last merge-pass, with 0 segments left of total size: 0 bytes
14/02/06 11:45:28 INFO mapred.LocalJobRunner: 
14/02/06 11:45:28 INFO mapred.Task: Task:attempt_local1238439569_0001_r_000000_0 is done. And is in the process of commiting
14/02/06 11:45:28 INFO mapred.LocalJobRunner: 
14/02/06 11:45:28 INFO mapred.Task: Task attempt_local1238439569_0001_r_000000_0 is allowed to commit now
14/02/06 11:45:28 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local1238439569_0001_r_000000_0' to hdfs://localhost.localdomain:8020/user/cloudera/output
14/02/06 11:45:28 INFO mapred.LocalJobRunner: reduce > reduce
14/02/06 11:45:28 INFO mapred.Task: Task 'attempt_local1238439569_0001_r_000000_0' done.
14/02/06 11:45:28 INFO mapred.JobClient:  map 100% reduce 100%
14/02/06 11:45:28 INFO mapred.JobClient: Job complete: job_local1238439569_0001
14/02/06 11:45:28 INFO mapred.JobClient: Counters: 26
14/02/06 11:45:28 INFO mapred.JobClient:   File System Counters
14/02/06 11:45:28 INFO mapred.JobClient:     FILE: Number of bytes read=7436
14/02/06 11:45:28 INFO mapred.JobClient:     FILE: Number of bytes written=199328
14/02/06 11:45:28 INFO mapred.JobClient:     FILE: Number of read operations=0
14/02/06 11:45:28 INFO mapred.JobClient:     FILE: Number of large read operations=0
14/02/06 11:45:28 INFO mapred.JobClient:     FILE: Number of write operations=0
14/02/06 11:45:28 INFO mapred.JobClient:     HDFS: Number of bytes read=5926764
14/02/06 11:45:28 INFO mapred.JobClient:     HDFS: Number of bytes written=0
14/02/06 11:45:28 INFO mapred.JobClient:     HDFS: Number of read operations=10
14/02/06 11:45:28 INFO mapred.JobClient:     HDFS: Number of large read operations=0
14/02/06 11:45:28 INFO mapred.JobClient:     HDFS: Number of write operations=4
14/02/06 11:45:28 INFO mapred.JobClient:   Map-Reduce Framework
14/02/06 11:45:28 INFO mapred.JobClient:     Map input records=24518
14/02/06 11:45:28 INFO mapred.JobClient:     Map output records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Map output bytes=0
14/02/06 11:45:28 INFO mapred.JobClient:     Input split bytes=129
14/02/06 11:45:28 INFO mapred.JobClient:     Combine input records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Combine output records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Reduce input groups=0
14/02/06 11:45:28 INFO mapred.JobClient:     Reduce shuffle bytes=0
14/02/06 11:45:28 INFO mapred.JobClient:     Reduce input records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Reduce output records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Spilled Records=0
14/02/06 11:45:28 INFO mapred.JobClient:     CPU time spent (ms)=0
14/02/06 11:45:28 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
14/02/06 11:45:28 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
14/02/06 11:45:28 INFO mapred.JobClient:     Total committed heap usage (bytes)=221126656
14/02/06 11:45:28 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
14/02/06 11:45:28 INFO mapred.JobClient:     BYTES_READ=2963382
[cloudera@localhost ~]$ 

The below is my MR code,

import java.io.IOException;
import java.util.*;
import java.text.SimpleDateFormat;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.util.*;

public class TestMR 
{
    public static class Map extends MapReduceBase implements Mapper<LongWritable,Text,Text,Text>
    { 
        public void map(LongWritable key, Text line, OutputCollector<Text, Text> output, Reporter reporter) throws IOException
        {
            final String [] split = line.toString().split(",");

            if(split[2].equals("Test"))
            {
                output.collect(new Text(split[0]), new Text(split[4] + "|" + split[7])); 
            }
        }
    }

    public static class Reduce extends MapReduceBase implements Reducer<Text,Text,Text,DoubleWritable>
    {
        public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, DoubleWritable> output, Reporter reporter) throws IOException
        {
            while(values.hasNext())
            {
                long t1=0, t2=0;
                SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

                String [] tmpBuf_1 = values.next().toString().split("|");
                String v1 = tmpBuf_1[0];
                try 
                {
                    t1 = df.parse(tmpBuf_1[1]).getTime();
                }
                catch (java.text.ParseException e) 
                {
                    System.out.println("Unable to parse date string: "+ tmpBuf_1[1]);
                    continue;
                }

                if(!values.hasNext())   
                    break;  

                String [] tmpBuf_2 = values.next().toString().split("|");       
                String v2 = tmpBuf_2[0];
                try 
                {
                    t2 = df.parse(tmpBuf_2[1]).getTime();
                }
                catch (java.text.ParseException e) 
                {
                    System.out.println("Unable to parse date string: "+ tmpBuf_2[1]);
                    continue;
                }      

                int vDiff = Integer.parseInt(v2) - Integer.parseInt(v1);    
                long tDiff = (t2 - t1)/1000;
                if(tDiff > 600)
                    break;

                double declineV = vDiff / tDiff;

                output.collect(key, new DoubleWritable(declineV));
            }
        }
    }

    public static void main(String[] args) throws Exception
    {
        JobConf conf = new JobConf(TestMR.class);
        conf.setJobName("TestMapReduce");
        conf.set("mapred.job.tracker", "local");

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(DoubleWritable.class);

        conf.setMapperClass(Map.class);
        conf.setCombinerClass(Reduce.class);
        conf.setReducerClass(Reduce.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        JobClient.runJob(conf);
    }
}

This is my first MapReduce program and I'm unable to locate the reason why it doesn't produce the output! Please let me know if there is any issue in my code or any better way of running the MapReduce job for getting output.

FYI, the testmr.jar file is in local file system and the CSV and output folders in HDFS.

1
  • Just to add: Please avoid using variable names that match keywords or function names Feb 13, 2014 at 8:37

1 Answer 1

3

If you look at the logs, you can see that the Map method isn't generating any output:

14/02/06 11:45:28 INFO mapred.JobClient:     Map input records=24518
14/02/06 11:45:28 INFO mapred.JobClient:     Map output records=0
14/02/06 11:45:28 INFO mapred.JobClient:     Map output bytes=0

As you can see, Map method is getting the input records, but it is producing 0 output records. So there must be something wrong with the logic in your Map Method:

final String [] split = line.toString().split(",");

        if(split[2].equals("Test"))
        {
            output.collect(new Text(split[0]), new Text(split[4] + "|" + split[7])); 
        }

I suggest that you test this logic as a simple Java code with some sample input data and make sure it works, then edit your MapReduce code and try running the job again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.