1

I have data in my Redshift cluster. I need to find the best and efficient way to delete the previously stored data when I re run the job.

I have these two column to determine previous data previous_key (column that corresponds to run_dt) and creat_ts (time when we load the data)

I found two approaches so far but they don't work in efficient way:

  • Use sql DELETE command – might be slow, eventually requires Vacuum the table to reclaim storage space and resort rows
  • Unload the data from a table into a file on S3 and then load table back (truncate and insert) with max clndr_key filtered out. Not really good either, might be risky.

Please suggest any good approach to rerun jobs on Redshift cluster. Note: partitions functionality is not available.

7
  • what is your problem with deleting / updating then running a vacuum and analyze? is the table always available? are you concerned about something? please elaborate.
    – Jon Scott
    Jun 24, 2019 at 5:22
  • Could you please clarify what you are actually trying to do? Are you just wanting to clear one column of data, or are you trying to delete selective rows of data? Jun 24, 2019 at 5:48
  • @johnRotenstein I want to delete selective rows based on load date. Jun 24, 2019 at 10:02
  • @jonscott We have large amount of data and performing delete operation would be slow. That's why I am looking for any other option, which can efficiently delete the data. yes tables would be always available. Jun 24, 2019 at 10:06
  • Can you tell us more about the data you are deleting? Is it, say, a whole day of data? How many days of data are stored in the table? Jun 24, 2019 at 10:15

2 Answers 2

1

Deleting data stored in Redshift with DELETE command will take time. The reason is that you are doing a soft delete, I mean you mark existing rows as deleted and then insert new row representing updated form of the data.

So one way is executing DELETE for junks of data. Instead of deleting one by one you should try to address multiple rows. Since each write takes place in 1 MB chunks of data, we should be minimizing those data read and writes eventually.

If you have a good information about the topology of the data stored in Redshift compute nodes and slices, addition to that information about distribution key and sort key, you can separate your DELETE command into multiple statements. (Any how we are expecting Redshift SQL Engine to do this for the SQL developer)

1

It sounds like you want to delete data after a certain time period.

In this case Redshift has a recommended approach "Time-Series Tables":


Basically, you create a new table for every insert of a fixed time window. Then the main interface to this data is a view that UNION's all of these tables together.

When you want to drop data after a time window, you can simply drop the entire table / remove it from the view definition. No Vacuum / Analyze / Expensive queries required.

Source: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-time-series-tables.html

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.