0

I write queries and procedures, I have no experience as a DB Admin and I am not in a position to be such. I work with hundreds of tables and certain older tables are difficult to work with. I suspect that statistics are a problem but the DBA states that isn't the case.

I don't know how to interpret statistics or even which ones I should look at. As an example, I am currently JOINing 2 tables, it is a simple JOIN that uses an index.

It returns just under 500 rows in 4 columns. It runs very quickly but not when in production with thousands of runs a day. My estimated and actual rows on this JOIN are off by 462%.

I have distilled this stored procedure down to a lot of very basic temp tables to locate the problem areas and it appears to be 2 tables, this example is one of them.

Estimate vs Actual

What I want is to know is which commands to run and what statistics to look at to take to the DBA to discuss the specific problem at hand. I do not want to be confrontational but informational. I have a very good professional relationship with this DBA but he is very black and white with his policies so I may not get anywhere with it in the end, but then I can also take that to my lead if I get stonewalled.

I ran a DBCC SHOW_STATISTICS on the table's index. I am not sure if this is the data I need or what I am really looking at. I would really like to know where to start with this. I have googled but all the pages I read are very geared towards DBAs and assume prior knowledge in areas I don't have.

Show_Statistics

Below is a sample of my JOIN obfuscated - my JOIN is on a temp table the first 2 conditions are needed for the Index, the date conditions when removed make the JOIN actually much worse with 10x the reads:

SELECT
    x.UniqueID,
    x.ChargeCode,
    x.dtDate,
    x.uniqueForeignID
INTO
    #AnotherTempTable
FROM
    Billing.dbo.Charges x
JOIN 
    #temptable y ON x.uniqueForeignID = y.uniqueID
                 AND x.ChargeCode = y.ChargeCode
                 AND @PostMonthStart <= x.dtDate 
                 AND x.dtDate < @PostMonthEnd

The JOIN above is part of a new plan where I have been dissecting all the data needed into temp tables to determine the root cause of the problem in high CPU and reads in production. Below is a list of all the statements that are executing, sorted by the number of reads. The second row is this example query but there are others with similar issues.

New Execution Plan

Below is the execution plan operations for the plan prior to my updates.

Old Exexcution Plan

While the new plan has better run time and closer estimates, I worry that I am still going to run into issues if the statistics are off. If I am completely off-base, please tell me and point me in the right direction, I will gladly bark up a different tree if I am making incorrect assumptions.

7
  • Have you tried updating statistics on a temporary table before the execution of your sample? Dec 26, 2019 at 16:07
  • @МаксимЗолотенко, I have created a Clustered Index on #temptable and updated the statistics on that table prior to the JOIN
    – DRT
    Dec 26, 2019 at 16:13
  • 1
    You've obfuscated too much/ supplied insufficient details. It isn't even clear why this cardinality estimate issue is a problem to you. Cardinality misestimates at the root of the plan that don't feed into any other operators are not an issue. It is not clear that these have anything to do with any performance issues you are experiencing. You should look at wait types your query is experiencing. You may be encountering blocking Dec 26, 2019 at 17:18
  • @MartinSmith, the stored procedure this is associated with runs very quickly when I test it and uses only 32 CPU and less than 10k reads on average. In production the proc on a given day ran 13.5k times in 11 hours and used 73.5 million CPU and did 7.3 billion reads. The only thing (from what I can see) wrong is that it is allocating memory incorrectly due to the estimates being off and it is using up server resources and causing CPU waiting times to increase - again I am not a DBA and I don't have access to most of the data, this is what I am supplied by the DBA
    – DRT
    Dec 26, 2019 at 17:54
  • 1
    yes, cardinality estimates can cause a problem - but the only code you have actually supplied they do not cause a problem. You have supplied an example where the execution plan selected was nested loops. It correctly estimates that 506 rows will be returned from the outer input and there will be 506 executions of the seek. The number of rows returned by the seek is fairly accurate but fewer are eliminated by the join operator than expected and this doesn't matter as there are no further operators that are affected by this (And the plan has no memory consuming operators at all) Dec 26, 2019 at 18:08

1 Answer 1

2

The first table returned shows some general information. You can see the statistics on this index were last updated 12/25/2019 at 10:19 PM. As of the writing of this answer, that is yesterday evening, so stats were updated recently. That is likely to be some kind of evening maintenance, but it could also be a threshold of data modifications that triggered an automatic statistics update.

There were 222,596,063 rows in the table at the time the statistics were sampled. The statistics update sampled 626,452 of these rows, so the sample rate is 0.2%. This sample size was likely the default sample rate used by a simple update statistics MyTable command.

A sample rate of 0.2% is fast to calculate but can lead to very bad estimates-- especially if an index is being used to support a foreign key. For example, a parent/child relationship may have a ParentKey column on the child table. A low statistics sample rate will result in very high estimates per parent row which can lead to strange decisions in query plans.

Look at the third table (the histogram). The RANGE_HI_KEY corresponds to a specific key value of the first column in this index. The EQ_ROWS column is the histogram's estimate of the number of rows that correspond to this key. If you get a count of the rows in this table by one of these keys in the RANGE_HI_KEY column, does the number in the EQ_ROWS column look like an accurate estimate? If not, a higher sample rate may yield better query plans.

For example, take the value 1475616. Is the count of rows for this key close to the EQ_ROWS value of 3893?

select count(*) from MyTable where FirstIndexColumn = 1475616

If the estimate is very bad, the DBA may need to increase the sample size on this table:

update statistics MyTable with sample 5 percent

If the DBA uses Ola Hallengren's plan (an excellent choice, in my opinion), this can be done by passing the @StatisticsSample parameter to the IndexOptimize procedure.

2
  • So this is a foreign key and I get fewer than 200 rows for the first several in the histogram. Accordingl,y what you are saying, is the sample size needs increased by the DBA to improve queries run on this table? All of our large transactional tables have problems with running slowly when joining, I have a feeling it has to do with the UPDATE STATISTICS that is running
    – DRT
    Dec 26, 2019 at 20:08
  • 1
    A higher sample rate will result in better statistics that should help the SQL optimizer generate a better query plan. In our application, we have some multi-billion row tables. We recommend to DBAs to get a 5% sample on these tables if they can. If they have a monster machine and can do 10%, that is even better, but diminishing returns cost too much to go beyond that. In my experience, specifying any sample size is better than SQL Server's default same rate for huge tables. Dec 26, 2019 at 20:49

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.