We're using SQL Server 2008 R2 Enterprise Edition.
We are measuring meteorological data from what we call MetMasts. Basically this is a mast with lots of equipment; anemometers (for wind speed) at different positions on the mast, thermometers , and air pressure. We measure every second.
And it takes up tooooo much disk space. The next generation of this equipment will generate over 10 GB per year each. And we’re going to have more than 1000 of these.
The current table design looks a bit like this:
CREATE TABLE #MetMast (
MetMastID INT NOT NULL IDENTITY(1,1),
MetMastName NVARCHAR(100),
CountryID INT,
InstallDate DATE
)
CREATE TABLE #MetMastData (
MetMastDataID BIGINT NOT NULL IDENTITY(1,1),
MetMastID INT NOT NULL,
MeasuredAt DATETIME2(0) NOT NULL,
Temperature REAL NULL,
WindSpeedAt10m REAL NULL,
WindSpeedAt30m REAL NULL,
AirPressure REAL NULL,
OneHundredMoreColumns VARCHAR(200),
CONSTRAINT PK_MetMastData PRIMARY KEY CLUSTERED
(
MetMastID ASC,
MeasuredAt ASC
))
WITH (DATA_COMPRESSION = ROW)
-- ON a file group, with table partitioning
ALTER TABLE #MetMastData WITH NOCHECK ADD CONSTRAINT FK_MetMast_MetMastID FOREIGN KEY (#MetMast) REFERENCES #MetMast(MetMastID)
The data is write once, read many, many times.
We use it in our data warehouse, where a typical question would be; Count how many times there is a 2 m/s difference between WindSpeedAt10m and WindSpeedAt30m when the temperature is above 20 degrees, per MetMast.
SELECT MetMastId, COUNT_BIG(*) FROM #metMastData
WHERE temperature>20 AND ABS(WindSpeedAt10m-WindSpeedAt30m) >2
GROUP BY MetMastID
In the future a tiny bit of data loss will be accepted.
We’re talking lossy compression of data here. I know we will have to define an acceptable error for each of the fields, as in 1% if we measure with 10% accuracy.
It worked for sound files (MP3 is quite big), so it might work for us as well.
But how is this done?
What table design should I go for?
How do I get started with lossy compression of data in database tables?
Best regards,
Henrik Staun Poulsen