1

I partition my data in hive based on a column value(date). So each date has it's own directory in /warehouse. right now I have about 240 dates, and a total of 70 million records evenly distributed across dates.

I also create another table containing the same data that does not has partitions.

When I query both table with the same queries, the partitioned table does not always out-perform the unpartitioned one. More specifically, partitioned table is slower when executing query with group by.

select count(*) from not_partitioned_table where date > '2018-07-27' and date < '2018-08-27

This took 22.146 seconds, and the count is 7427366.

select count(*) from partitioned_table where date > '2018-07-27' and date < '2018-08-27

This took 22.723 seconds, and also returns 7427366 for count.

However when group by is added, partitioned table perform worse than un-partitioned table.

select count(*) from not_partitioned_table where dated > '2018-07-27' and date < '2018-08-27' group by col_name; 

This took 39.733 seconds and 26,724 rows were returned.

select count(*) from partitioned_table where dated > '2018-07-27' and date < '2018-08-27' group by col_name; 

This took 76.648 seconds seconds and 26,724 rows were returned.

Why is the partitioned table slower in this scenario?

EDIT

This is how I create my partitioned table:

CREATE TABLE all_ads_from_csv_partitioned3(
id STRING,
...
)
PARTITIONED BY(datedecoded STRING)
STORED AS ORC;

And under 2018-10-08 15:34 /warehouse/tablespace/managed/hive/partitioned_table/, there are 240 directories(240 partitions), each has the format of /warehouse/tablespace/managed/hive/partitioned_table/dated='the partitioned date', and each partition contains roughly 10 buckets.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.