1

I have a oozie workflow. I am submitting all the hive actions with

<name>mapred.job.queue.name</name>
<value>${queueName}</value>

But for few hive actions, the job launched is not in specified queue; it is invoked in default queue.

Please suggest me the cause behind this behavior and solution.

3 Answers 3

7

A. Oozie specifics Oozie propagates the "regular" Hadoop properties to a "regular" MapReduce Action.

But for other types of Action (Shell, Hive, Java, etc.) where Oozie runs a single Mapper task in YARN, it does not consider that it's a real MapReduce job. Hence it uses a different set of undocumented properties always prefixed with oozie.launcher. Look into that post for example.

So in your case the actual property to set would be oozie.launcher.mapred.job.queue.name

B. TEZ specifics Excerpt from HortonWorks documentation:

For example, in Hive you can use the tez.queue.name property in hive-site.xml to specify the queue to use for Hive-on-Tez jobs. To assign Hive-on-Tez jobs to use the "engineering" queue, add the following property to hive-site.xml:

<property>  <name>tez.queue.name</name> <value>engineering</value> </property>
2
  • To make it clear: the "regular" Hadoop properties will be propagated to any MapReduce job started by Hive. But the main Hive session, and any job run in Local Mode, will use the "launcher" properties (or the default props failing that) Sep 7, 2015 at 14:21
  • Thanks for the answer Samson ! But even after setting 'oozie.launcher.mapred.job.queue.name' the job is running in default queue. Actually, I am using TEZ as execution engine. Is the property not applying for TEZ job ?
    – Anup
    Sep 8, 2015 at 5:25
0

In my case using Oozie it seems that there is no way to add --hiveconf tez.queue.name=myqueue in an xml tag. Trying also by updating the hive-site.xml on hdfs and still no usage of the defined queue. I've end up with updating the .hql file by adding the below at the beginning of the required query:

# in your hql file
SET tez.queue.name=${myqueue};
0

This is an extension to what Mathieu mentioned. It worked for me when I hard coded the queue name in below portion -

#in your hql file

SET tez.queue.name=queue name;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.