15

I have successfully created and added Dynamic partitions in an Internal table in hive. i.e. by using following steps:

1-created a source table

2-loaded data from local into source table

3- created another table with partitions - partition_table

4- inserted the data to this table from source table resulting in creation of all the partitions dynamically

My question is, how to perform this in external table? I read so many articles on this, but i am confused , that do I have to specify path to the already existing partitions for creating partitions for external table??

example: Step 1:

create external table1 ( name string, age int, height int)
location 'path/to/dataFile/in/HDFS';

Step 2:

alter table table1 add partition(age) 
location 'path/to/already/existing/partition'

I am not sure how to proceed with partitioning in external tables. Can somebody please help by giving step by step description of the same?.

Thanks in advance!

1
  • yes you need to specify the path. Oct 24, 2018 at 9:36

4 Answers 4

50

Yes, you have to tell Hive explicitly what is your partition field.

Consider you have a following HDFS directory on which you want to create a external table.

/path/to/dataFile/

Let's say this directory already have data stored(partitioned) department wise as follows:

/path/to/dataFile/dept1
/path/to/dataFile/dept2
/path/to/dataFile/dept3

Each of these directories have bunch of files where each file contains actual comma separated data for fields say name,age,height.

e.g.
    /path/to/dataFile/dept1/file1.txt
    /path/to/dataFile/dept1/file2.txt

Now let's create external table on this:

Step 1. Create external table:

CREATE EXTERNAL TABLE testdb.table1(name string, age int, height int)
PARTITIONED BY (dept string)
ROW FORMAT DELIMITED
STORED AS TEXTFILE
LOCATION '/path/to/dataFile/';

Step 2. Add partitions:

ALTER TABLE testdb.table1 ADD PARTITION (dept='dept1') LOCATION '/path/to/dataFile/dept1';
ALTER TABLE testdb.table1 ADD PARTITION (dept='dept2') LOCATION '/path/to/dataFile/dept2';
ALTER TABLE testdb.table1 ADD PARTITION (dept='dept3') LOCATION '/path/to/dataFile/dept3';

Done, run select query once to verify if data loaded successfully.

2
  • 2
    Thanks for such great explanation Sachin!! Just one more thing. This will be a static partitioning right? How to do it dynamically? Sep 15, 2015 at 14:45
  • Can we do msck repair table insted of doing Step 2.. In case we have too many partitions.
    – Andy_101
    Oct 15, 2020 at 15:57
5

1. Set below property

set hive.exec.dynamic.partition=true

set hive.exec.dynamic.partition.mode=nonstrict

2. Create External partitioned table

create external table1 ( name string, age int, height int) location 'path/to/dataFile/in/HDFS';

3. Insert data to partitioned table from source table.

Basically , the process is same. its just that you create external partitioned table and provide HDFS path to table under which it will create and store partition.

Hope this helps.

2
  • it definitely helped pradeep!! works like a charm!! got to underdtand both static and dynamic partitions in external tables!!! cheers!! Sep 15, 2015 at 16:36
  • 5
    where is partitions in your external table table1? May 21, 2019 at 13:28
0

The proper way to do it.

  1. Create the table and mention it is partitioned.

    create external table1 ( name string, age int, height int) partitioned by (age int) stored as ****(your format) location 'path/to/dataFile/in/HDFS';

  2. Now you have to refresh the partitions in the hive metastore.

    msck repair table table1

This will take care of loading all your partitions into the hive metastore.

You can use msck repair table at any point during your process to have the metastore updated.

-1

Follow the below steps:

  1. Create a temporary table/Source table

    create table source_table(name string,age int,height int) row format delimited by ',';
    

    Use your delimiter as in the file instead of ',';

  2. Load data into the source table

    load data local inpath 'path/to/dataFile/in/HDFS';
    
  3. Create external table with partition

    create external table external_dynamic_partitions(name string,height int) 
    partitioned by (age int) 
    location 'path/to/dataFile/in/HDFS';
    
  4. Enable dynamic partition mode to nonstrict

    set hive.exec.dynamic.partition.mode=nonstrict
    
  5. Load data to external table with partitions from source file

    insert into table external_dynamic partition(age) 
    select * from source_table;
    

That's it. You can check the partitions information using

show partitions external_dynamic;

You can even check if it is an external table or not using

describe formatted external_dynamic;

External table is a type of table in Hive where the data is not moved to the hive warehouse. That means even if U delete the table, the data still persists and you will always get the latest data, which is not the case with Managed table.

1
  • Should the source_table be external, since it's loaded from local files?
    – Wassadamo
    Jun 18, 2020 at 20:10

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.