I am working with hive on an external table in text format. I populate this table every hour, but I partition the table by month (The dataset is relatively small). Each hour I want to insert new data into some partitions.
The INSERT INTO clause results in the creation of a new file in the existing partition which contains the old data. This way in the end of the month i will have around 700 small files in each partition.
Is there a way for HIVE to append the data to the old file in the partition (Without using UNION ALL on the old data)?