0

On our cluster, even the simplest job "wordcount" is running in local mode. But there are previous jobs run using oozie, we observe that they run in cluster mode. Is there a conflict of native hadoop jobs with the hadoop jobs run using oozie? Here is the error stack trace-

java.lang.IllegalArgumentException: Can't read partitions file
    at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:111)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:408) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:825) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1480) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475) at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:296) at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:82)

Our job is running fine on a test cluster where we do not have Oozie installed. Hence not sure if the two things are related. Thanks in advance.

We are using Cloudera distribution.

1
  • If this helps, I have more details. Apparently, we are submitting the jobs locally. Hadoop is not configured to run the jobs submitted locally. So the job tracker is not running the jobs. So we are probably needed to change the following parameters in the job conf. mapred.job.tracker = <machinename>:8021 ; fs.default.name = hdfs://<machinename>:8020 ; hadoop.job.ugi", "hdfs:hadoop using conf.set in our code. I would like to keep the code unchanged and not hardcode things, rather would prefer a solution to modify hadoop conf files. Any help is greatly appreciated.
    – Sumod
    Aug 22, 2012 at 14:41

2 Answers 2

0

It really sounds like they're using different configs. I'd start by making sure that everything you're running is using the same HADOOP_HOME.

0

After many retries, we have now switched to BigInsights. I personally think CDH is a great Hadoop flavor. But as we had not done the original installation, the overhead of reliance was too much for our project. As we had success earlier with BigInsights, we hope to replicate it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.