When I using Spark HiveContext to do sql like insert overwrite a select * from b
, at last, there are many small files(400+) on the table's corresponding directory of HDFS, many of them are empty files. So, I try to use coalesce
to reduce the file numbers, sample codes is:
val df = hiveContext.sql("insert overwrite a select * from b")
df.coalesce(50).collect
But the output files are still 400+, looks like coalesce
doesn't work.
Can someone help on this?