3

Can someone speak to 'common practice' or general "acceptable practice" on handling data that is highly transient for short periods of time - yet has to have historical presence for years...

Take for instance a small tour company - booking tours. The passenger manifest would be highly fluid, up until the tour.. once the tour completed - the data is technically obsolete, but useful for reporting or trending...

With HIGH volume - the 'search' or 'creation' of a tour - the db would become THICK with data that is only going to change rarely.

Is it common to have tables with similar structure - that "move" data from one state to another (PRE/POST event)... as say an interim before data warehousing it to a semi-flat structure for 'pure' reporting

Is that advisable, common or correct? Is there a better way to do that... or is dome DBA gonna come in and go "WTF were you thinking"

1
  • What you are looking for is "data warehouse".
    – jva
    Jul 23, 2012 at 18:52

2 Answers 2

1

Standard would be to have some sort of flag to indicate record obsolescence. There are several methods available that would allow you to deal well with performance issue such as pruning and indexing techniques. These would probably also include some kind of archiving strategy. You would likely move old data from the table (how often and when would depend on performance requirements) either with some sort of bulk insert into a history table with the same structure (how you actually implement this would depend on your DBMS but choose the most robust method) or better with some sort or partitioning strategy if your DBMS has a robust partitioning system. You also might think about some sort of multi-database archiving strategy if the analyst who need the historic data are not the same as the people who run the operational systems.

0

Of course, such a design decision depends on other factors. In general, though, it is not a good idea to move data from one table to another.

A better option is to have effective and end dates on your records. So, if someone signs up for a tour, then their record starts with an effdate of that date. If they drop out, then that person gets and end date. If they sign up again, they get a new record, with a new eff date.

This allows you to reconstruct the history at any point in the past.

2
  • yes but don't you end up with gobbs of data the db engine has to process on a query. Let's get crazy and say this is an airline or some such entity with 1000's of passengers (records) on 1000's of flights per day.... even if you filter on date - isn't it?? If a clerk goes to look up x person on x flt - and you have 2 years worth of data - most of which ISN'T relevant - that sees very inefficient on the db side of things
    – j-p
    Jul 23, 2012 at 15:51
  • You would query relevant data by index and that is O(logN).
    – jva
    Jul 23, 2012 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.