1

I just configured my Ubuntu 13.10 to work in pseudo-distributed mode for my mapreduce code development. I had installed hadoop 0.20.2 version of hadoop. Everything sis running fine and I am able to start all five deamons as well.

On same machine I had downloaded eclipse and added all hadoop based libraries into it. I am able to run my map reduce word count example also from eclipse IDE directly. Only thing which is bothering me is that while I run my word count example it prints in console something like this:

13/09/23 16:11:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your  
platform... using builtin-java classes where applicable
13/09/23 16:11:05 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See    
JobConf(Class) or JobConf#setJar(String).
13/09/23 16:11:05 INFO input.FileInputFormat: Total input paths to process : 1
13/09/23 16:11:06 INFO mapred.JobClient: Running job: job_local_0001
13/09/23 16:11:06 INFO util.ProcessTree: setsid exited with exit code 0
13/09/23 16:11:06 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c931fc
13/09/23 16:11:06 INFO mapred.MapTask: io.sort.mb = 100
13/09/23 16:11:07 INFO mapred.JobClient:  map 0% reduce 0%
13/09/23 16:11:07 INFO mapred.MapTask: data buffer = 79691776/99614720
13/09/23 16:11:07 INFO mapred.MapTask: record buffer = 262144/327680
13/09/23 16:11:08 INFO mapred.MapTask: Starting flush of map output
13/09/23 16:11:08 INFO mapred.MapTask: Finished spill 0
13/09/23 16:11:08 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the 
process of commiting
13/09/23 16:11:09 INFO mapred.LocalJobRunner: 
13/09/23 16:11:09 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
13/09/23 16:11:09 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1342ba4
13/09/23 16:11:09 INFO mapred.LocalJobRunner: 
13/09/23 16:11:09 INFO mapred.Merger: Merging 1 sorted segments
13/09/23 16:11:10 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total 
size: 48 bytes
13/09/23 16:11:10 INFO mapred.LocalJobRunner: 
13/09/23 16:11:10 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the 
process of commiting
13/09/23 16:11:10 INFO mapred.LocalJobRunner: 
13/09/23 16:11:10 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
13/09/23 16:11:10 INFO output.FileOutputCommitter: Saved output of task 
'attempt_local_0001_r_000000_0' to outputWords
13/09/23 16:11:10 INFO mapred.JobClient:  map 100% reduce 0%
13/09/23 16:11:12 INFO mapred.LocalJobRunner: reduce > reduce
13/09/23 16:11:12 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
13/09/23 16:11:12 WARN mapred.LocalJobRunner: job_local_0001

java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:284)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
 Exception in thread "Thread-1" java.lang.NoClassDefFoundError:    
org/apache/commons/httpclient/HttpMethod
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:300)
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 1 more
13/09/23 16:11:13 INFO mapred.JobClient:  map 100% reduce 100%
13/09/23 16:11:13 INFO mapred.JobClient: Job complete: job_local_0001
13/09/23 16:11:13 INFO mapred.JobClient: Counters: 20
13/09/23 16:11:13 INFO mapred.JobClient:   File Output Format Counters 
13/09/23 16:11:13 INFO mapred.JobClient:     Bytes Written=42
13/09/23 16:11:13 INFO mapred.JobClient:   FileSystemCounters
13/09/23 16:11:13 INFO mapred.JobClient:     FILE_BYTES_READ=534
13/09/23 16:11:13 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=63640
13/09/23 16:11:13 INFO mapred.JobClient:   File Input Format Counters 
13/09/23 16:11:13 INFO mapred.JobClient:     Bytes Read=63
13/09/23 16:11:13 INFO mapred.JobClient:   Map-Reduce Framework
13/09/23 16:11:13 INFO mapred.JobClient:     Map output materialized bytes=52
13/09/23 16:11:13 INFO mapred.JobClient:     Map input records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce shuffle bytes=0
13/09/23 16:11:13 INFO mapred.JobClient:     Spilled Records=8
13/09/23 16:11:13 INFO mapred.JobClient:     Map output bytes=110
13/09/23 16:11:13 INFO mapred.JobClient:     Total committed heap usage (bytes)=231350272
13/09/23 16:11:13 INFO mapred.JobClient:     CPU time spent (ms)=0
13/09/23 16:11:13 INFO mapred.JobClient:     SPLIT_RAW_BYTES=124
13/09/23 16:11:13 INFO mapred.JobClient:     Combine input records=12
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce input records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce input groups=4
13/09/23 16:11:13 INFO mapred.JobClient:     Combine output records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient:     Reduce output records=4
13/09/23 16:11:13 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
13/09/23 16:11:13 INFO mapred.JobClient:     Map output records=12

In above output if you see there are few things I am not sure are correct:

  1. its printing mapred.JobClient: Mapred is old library of hadoop so how can i make it to mapreduce ( alreday added recent new library into eclipse still getting same mapred message)
  2. Why is this error coming : java.lang.NoClassDefFoundError

I am able to get output directory generated with proper results also.

Let me know if need any other details.

Hope to get an answer.

Happy hadooping!!!

1
  • Already used Job for setting configuration part instead JobConfig in mapreduce code. Sep 23, 2013 at 23:21

1 Answer 1

1

You are getting --

Exception in thread "Thread-1" java.lang.NoClassDefFoundError:    
org/apache/commons/httpclient/HttpMethod

because, you haven't included some dependent jars in your classpath.

Try including the following too located inside your lib/ directory and retry --

commons-httpclient-3.1.jar
commons-cli-1.2.jar
commons-logging-1.0.4.jar
commons-logging-api-1.0.4.jar
log4j-1.2.15.jar
commons-cli-1.2.jar
jackson-core-asl-1.5.2.jar
jackson-mapper-asl-1.5.2.jar

If including these doesn't work, please include all the jars in the lib/ directory.

Furthermore, mapred.JobClient is not deprecated and referenced by Hadoop (both mapred API or mapreduce API).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.