I have data in my Redshift cluster. I need to find the best and efficient way to delete the previously stored data when I re run the job.
I have these two column to determine previous data previous_key (column that corresponds to run_dt) and creat_ts (time when we load the data)
I found two approaches so far but they don't work in efficient way:
- Use sql DELETE command – might be slow, eventually requires Vacuum the table to reclaim storage space and resort rows
- Unload the data from a table into a file on S3 and then load table back (truncate and insert) with max clndr_key filtered out. Not really good either, might be risky.
Please suggest any good approach to rerun jobs on Redshift cluster. Note: partitions functionality is not available.