We had an issue with our ingestion process that would result in partitions being added to a table in Hive, but the path in HDFS didn't actually exist. We've fixed that issue, but we still have these bad partitions. When querying these tables using Tez, we get FileNotFound exception, pointing to the location in HDFS that doesn't exist. If we use MR instead of Tez, the query works (which is very confusing to me), but it's too slow.
Is there a way to list all the partitions that have this probem? MSCK REPAIR
seems to handle the opposite problem, where the data exists in HDFS but there is no partition in Hive.
EDIT: More info. Here's the output of the file not found exception:
java.io.FileNotFoundException: File hdfs://<server>/db/tables/2016/03/14/mytable does not exist.
If I run show partitions <db.mytable>
, I'll get all the partitions, including one for dt=2016-03-14
.
show table extended like '<db.mytable>' partition(dt='2016-03-14'
returns the same location:
location:hdfs://server/db/tables/2016/03/14/mytable
.
location:hdfs://server/db/tables/2017/06/07/mytable
. The only difference is that it actually exists. – Andrew Jul 6 '17 at 14:08