8

I'm trying to merge 2 datasets, say A and B. The dataset A has a variable "Flag" which takes 2 values. Rather than jut merging both data together I was trying to merge 2 datasets based on "flag" variable.

The merging code is the following:

create table new_data as
select a.*,b.y
from A as a left join B as b
on a.x=b.x

Since I'm running Hive code through CLI, I'm calling this through the following command

hive -f new_data.hql

The looping part of the code I'm calling to merge data based on "Flag" variable is the following:

for flag in 1 2;
do
  hive -hivevar flag=$flag -f new_data.hql
done

I put the above code in another ".hql" file asn calling it:

hive -f loop_data.hql

But it's throwing error.

cannot recognize input near 'for' 'flag' 'in'

Can anybody please tell me where I'm making mistake.

Thanks!

6
  • 1
    The looping code is not a hive query/script. Hive -f can only run hive queries. Use the looping in a shell script .
    – K S Nidhin
    Commented Feb 25, 2016 at 20:16
  • @KSNidhin: Thanks a lot for the comment. I'm running the hive code through putty. Could you please explain what do you mean by shell script? I'm technologically bit novice.
    – Beta
    Commented Feb 25, 2016 at 20:36
  • the flag variable has 2 values 1 and 2 right ?Which is a filter criteria in the hive query right ?.
    – K S Nidhin
    Commented Feb 25, 2016 at 20:41
  • @KSNidhin: You are right. Flag has 2 variable and it is the filter criteria. The idea is break the data by flag and merge the data for each of these partition.
    – Beta
    Commented Feb 25, 2016 at 20:45
  • then you could add the filter criteria in the hive query and do the merging in the query itself and then run the query using hive -f <> alone.
    – K S Nidhin
    Commented Feb 25, 2016 at 20:48

1 Answer 1

10
  1. You should be adding the loop logic to shell script.

File Name: loop_data.sh

for flag in 1 2;
do
  hive -hivevar flag=$flag -f new_data.hql
done

And execute the script like:

sh loop_data.sh
  1. In your new_data.hql script, you are creating table. Since you should split out the DDL & DML in 2 separate scripts. Like

DDL: create_new_data.hql

create table new_data as
select 
  a.*,
  b.y
from 
  A as a left join 
  B as b on 
  a.x = b.x
where 
  1 = 0;

DML: insert_new_data.hql

insert into new_data 
select 
  a.*,
  b.y
from 
  A as a left join 
  B as b on 
  a.x = b.x
where
  flag = ${hiveconf:flag}

And update you shell script like:

File Name: loop_new_data.sh

# Create table
hive -f create_new_data.hql

# Insert data
for flag in 1 2;
do
  hive -hiveconf flag=$flag -f insert_new_data.hql
done

And execute it like:

sh loop_new_data.sh

Let me know if you want more info.

5
  • Thanks Ambish for your reply!
    – Beta
    Commented Mar 23, 2016 at 15:40
  • how can i used it from hive shell Commented Sep 9, 2016 at 10:06
  • @sahildesai please elaborate your question.
    – Ambrish
    Commented Sep 9, 2016 at 10:09
  • 4
    the above solution is for command line but i am using hive queries on hive shell Commented Sep 9, 2016 at 10:19
  • Although @Beta 's problem has been solved, still the topic has no answer: how to loop in hive? not from shell but directly in hive?
    – Dr Potato
    Commented May 5, 2021 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.