1

I currently have tables that are partitioned out by year & month for our sales transactions. For example, we have sales tables that would look something like this:

  • factdailysales_201501
  • factdailysales_201502
  • factdailysales_201503 etc ...

Generally, I've always performed dynamic SQL to capture a Start Date, End Date, find out what partitions those are, and then loop through each of those partitions ... but its starting to become such a hassle and I've learned that this is probably not the best way to do it in terms of just maintenance, trouble shooting, and performance.

I decided to build a view that would UNION ALL of my sales partitions together. However, I don't want selecting from the view to have to scan all of the partitions on execution, it would take away the whole purpose of partitioning tables out. Because of this, I added check constraints on date to each of my sales tables. This way when I selected from the view, it would know which tables to access from instead of scanning every table.

Here are the following examples below:

    SELECT SUM([retail])  
    FROM Sales_Orig 
    WHERE [Date] >= '2015-03-01' 

This query has the execution plan of only pulling from the partitions that I need.

My problem that i'm facing right now is that most of the time when my team will be writing stored procedures, they would more than likely write their queries where a date variable is passed into the where statement.

DECLARE @SD DATE = '2015-03-01' 

SELECT SUM([retail])  
FROM Sales_Orig 
WHERE [Date] >= @SD 

However, when a variable is being passed in, the execution plan now scans ALL of the partitions in the view, causing the performance to take wayyy longer than when I hard coded in the date

I suppose I could do dynamic SQL again and insert the date string into the SELECT statement, but it would bring me back to the beginning of trying to get rid of dynamic SQL in the first place for this simple sales query.

So my question is, am I setting this up wrong? Am I on the right track? It seems that the view can't take in a variable for the check constraint and ends up scanning every table. Is there another approach anyone would recommend? Maybe my original solution of just looping through partitions via dynamic SQL is the best way to do it?

** EDIT ** http://sqlsunday.com/2014/08/31/partitioned-views/

This article is actually where I initially saw the idea! It seems when using that exact same solution, I'm still experiencing the same struggle!

Thanks!!

2
  • Well I'm no expert. To me, I've heard of partitioning tables as in having the Enterprise edition SQL Server have different files so you can , but having different tables doesn't sound like the best system. I think having one big table with a clustered index on the date column would perform better. That way when you query, it scans the index and looks at only the table section it needs to. So letting SQL pick the section as it was designed to instead of forcing to with dynamic SQL
    – Stephan
    May 8, 2015 at 18:15
  • 1
    @Stephan Hi Stephan. Agreed, having multiple tables partitioned out isn't always ideal, but there are instances where it's pretty beneficial from my experience. With that said, it's simply the current way everything is set up, and to change the architectural design now would be out of the question :(. May 8, 2015 at 18:20

2 Answers 2

0

Okay this might work. It's a table-valued function that only access tables according to your @start and @end parameters so only accessing your "partitions" that it needs. I figured you could take this concept and write some dynamic SQL to create all the if statements.

Now of course new tables are added every day so how does that tie in. Well I think the best way would be is that every day you alter the function adding the next sales table. That way querying it is simple. And you could use the same dynamic sql you used to create the function to alter it which should be relatively simple.

Note: I added default values that are the min and max of the data type DATE. That way you could query something like everything from 20140101 and onward or vice versa.

Your tables

SELECT CAST('20150101' AS DATE) datesVal INTO factDailySales_20150101;
SELECT CAST('20150102' AS DATE) datesVal INTO factDailySales_20150102;
SELECT CAST('20150103' AS DATE) datesVal INTO factDailySales_20150103;

The Function

CREATE FUNCTION ufn_factTotalSales (@Start DATE = '17530101', @End DATE = '99991231')
RETURNS @factTotalSales TABLE
(
    datesVal DATE
)
AS
BEGIN
    IF(CAST('20150101' AS DATE) BETWEEN @Start AND @End)
    BEGIN
        INSERT INTO @factTotalSales
            SELECT datesVal
            FROM factDailySales_20150101
    END
    IF(CAST('20150102' AS DATE) BETWEEN @Start AND @End)
    BEGIN
        INSERT INTO @factTotalSales
            SELECT datesVal
            FROM factDailySales_20150102
    END
    IF(CAST('20150103' AS DATE) BETWEEN @Start AND @End)
    BEGIN
        INSERT INTO @factTotalSales
            SELECT datesVal
            FROM factDailySales_20150103
    END
    RETURN;
END
GO

All tables

SELECT *
FROM ufn_factTotalSales(default,default)

All tables greater than or equal to 20150102

SELECT *
FROM ufn_factTotalSales('20150102',default)

**All tables less than or equal to 20150102

SELECT *
FROM ufn_factTotalSales(default,'20150102')

All tables between specific range

SELECT *
FROM ufn_factTotalSales('20150101','20150102')

Is this the ideal solution? No. The ideal would be to combine all tables into one and having good indexes. I know you said that wouldn't work because of the way other code has been written. Hear me out. Now perhaps this is off the wall, lets say you do combine the tables but obviously there are old scripts looking for specific daily sales tables. Maybe you could create views with the dailySales names that access the factTotalSales. OR You could create synonyms for the factTotalSales that would correspond to each factDailySales.

Maybe you could look into that. It wouldn't be easy, but I think letting SQL Server optimize your queries the way it was designed is a better way of doing it instead of forcing it with dynamic SQL.

Just my two cents. Hope this helps. At the very least, I hope it gave you some ideas.

2
  • Thank you for your advice! I've definitely considered that before, but the reason I strayed away from a tbl valued fn was because of tbl variable performance. Its to my understanding that tbl variables become a very inefficient when storing large amounts of rows, especially if we are joining. I guess my biggest question is, my current view works amazing and is actually pretty darn fast when I physically pass in a date value instead of a @date variable. It seems the issue is that CHECK constraint doesn't take in variables for some reason. Unless it can, and I'm setting it up incorrectly? May 8, 2015 at 19:44
  • Yeah I just realized table-valued functions might not be able to use indexes efficiently. Even so, I would test it out with your data. It doesn't really hurt to try. And sadly views don't allow input variables. Now that I think about it, if you have good indexes then your where clause should only have to do a quick index scans.
    – Stephan
    May 8, 2015 at 20:04
0

5 years later: option(recompile).

The planner needs to have access to the constants to eliminate the table entirely from the query plan. With a variable, without a forced recompile, a generic plan is used. (Related: parameter sniffing.)

While this means the query plan is larger as it has to include all tables, it does not mean that all tables are actually scanned: look at the IO stats, as table scan elimination occurs even if such shows in the query plan.

The 'Number Of Executions' in the query plan will be 0 when the tables are not scanned: unfortunately, these branches are still reported as a non-zero percentage cost "Table Scan" node in the query plan & UI, which will appear high proportionally if the query is trivially fast. The displayed percentage cost of these extra "Table Scan" nodes approaches zero as the amount of data returned from the actually used base tables increases.

This same optimization/elimination occurs when the view is not a Partitioned View (eg. base tables are missing partition column in PK), yet the underlying tables have a suitable Check Constraint on the filtered column. It also occurs when the view selects a constant value to establish the partition that is not otherwise stored in the table. With a constant in the query or recompiled plan the tables will be eliminated entirely. With a variable the tables will still not actually be scanned and thus eliminated logically during query execution.

The use of a proper Partitioned View is only really beneficial to allow a direct Insert & Update, with the major caveat that it requires the partition column to be in each table's PK and disallows the use of an identity column (making a Partitioned View largely useless IMOHO). SQL Server handles the optimizations very similarly for other quasi-Partitioned View cases.

(This is on SQL Server 2014; earlier versions might not have optimized the different patterns as efficiently.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.