0

I am running MSCK REPAIR TABLE table name on a hive table which contains more than 28K partitions and we'll be receiving a partition for every 10 mins.

When we run MSCK REPAIR TABLE on this table every 10 mins it's taking a lot if time

Could someone suggest why it's taking more time? (i.e more than 5-10 mins)

Thanks in advance.

Hive version: 1.1.0

1 Answer 1

2

This is a very bad practice.
Whether a directory is already mapped to a partition or not, MSCK REPAIR still needs to get the list of all directories as well as the list of all partitions and compare them.
Instead, you should use alter table ... add partition ... to add a partition every time you add a directory.

2
  • Thanks @Dudu Markovitz for quick reply , but we're ingesting data to HDFS via Flume and built an external table on top of the directory where we place the data from Flume
    – Naga
    Jun 17, 2017 at 17:17
  • 1
    @Naga, I don't get the issue. While you creating the directory can't you also add the partition or at least store the name of the directory somewhere and add it later as a partition? Jul 4, 2017 at 7:11

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.