0

When I using Spark HiveContext to do sql like insert overwrite a select * from b, at last, there are many small files(400+) on the table's corresponding directory of HDFS, many of them are empty files. So, I try to use coalesce to reduce the file numbers, sample codes is:

val df = hiveContext.sql("insert overwrite a select * from b")
df.coalesce(50).collect

But the output files are still 400+, looks like coalesce doesn't work.

Can someone help on this?

8
  • How are you verifying that coalesce doesn't work ? Because as I see you are doiing a collect after this !
    – Shivansh
    Nov 15, 2016 at 6:26
  • collect is used to trigger the job. Since the job is an insert operation, so there is very little data will be 'collected' to the driver. The data is still written to the HDFS!
    – Tom
    Nov 15, 2016 at 6:28
  • I am not able to understand coalesce is just a shuffle transformation , with every action it will be recalculated ! Can you add more code here the one having the insertion ?
    – Shivansh
    Nov 15, 2016 at 6:30
  • Thanks @ShivanshSrivastava for the reply. More code? I think the code above is enough to illustrate the problem..What code do you want to know?
    – Tom
    Nov 15, 2016 at 6:39
  • what in your code should result with "the output" ("output files 400+")?
    – Yaron
    Nov 15, 2016 at 6:43

1 Answer 1

3

Your example will not merge output files because coalesce is done after executing SQL with insert into and on this insert into results (which I suppose is an empty Dataframe).

Try rewriting code to something like that:

hiveContext.sql("select * from b").coalesce(50).write.mode("overwrite").saveAsTable("a")
2
  • Thanks @Marisz. One quick question is how to specify partition for table a.That is, I want to write into one partition of table a.There is a method partitionedBy on DataFrameWriter,but the method doc doesn't look like working with Hive Partition.
    – Tom
    Nov 15, 2016 at 8:09
  • partitionBy works as expected when writing to Hive, I've just tested it on spark 2.0
    – Mariusz
    Nov 15, 2016 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.