83

I develop software that stores a lot of data in one of its database tables (SQL Server version 8, 9 or 10). About 100,000 records are inserted into that table per day. This is about 36 million records per year. For performance, I create a new table every day (a table with the current date in its name) to lower the number of records per table.

Was this a good idea? Is there a record limit for SQL server tables? Or do you know how many records (more or less) can be stored in a table before performance is lowered significantly?

1
  • 2
    True words, wrong place. One indicator whether a part of a program is "critical" or not, is, if its execution can significantly influence the performance of my application. So while it might be true that devs worry about the wrong things most of the time (I dont know), this one here, is definitively not one of those wrong things.
    – Moss
    Feb 9, 2021 at 10:14

12 Answers 12

98

These are some of the Maximum Capacity Specifications for SQL Server 2008 R2

  • Database size: 524,272 terabytes
  • Databases per instance of SQL Server: 32,767
  • Filegroups per database: 32,767
  • Files per database: 32,767
  • File size (data): 16 terabytes
  • File size (log): 2 terabytes
  • Rows per table: Limited by available storage
  • Tables per database: Limited by number of objects in a database
4
  • 26
    I would suspect that if you have more than 9,223,372,036,854,775,807 rows you would run into problems though (maximum size of a bigint) Mar 22, 2011 at 13:41
  • 12
    Have you ever computed the number of years it would take to get to that row count at the 100000 rows/day the OP mentioned ? Dec 31, 2011 at 0:08
  • 1
    Only to store that amount of indexes you already need 67,108,864 terabytes. Jul 11, 2014 at 7:49
  • From 2008R2, Enterprise Editions can hold Petabytes of data blog.sqlauthority.com/2013/07/21/… Feb 10, 2016 at 14:45
64

I have a three column table with just over 6 Billion rows in SQL Server 2008 R2.

We query it every day to create minute-by-minute system analysis charts for our customers. I have not noticed any database performance hits (though the fact that it grows ~1 GB every day does make managing backups a bit more involved than I would like).

Update July 2016

Row count

We made it to ~24.5 billion rows before backups became large enough for us to decide to truncate records older than two years (~700 GB stored in multiple backups, including on expensive tapes). It's worth noting that performance was not a significant motivator in this decision (i.e., it was still working great).

For anyone who finds themselves trying to delete 20 billion rows from SQL Server, I highly recommend this article. Relevant code in case the link dies (read the article for a full explanation):

ALTER DATABASE DeleteRecord SET RECOVERY SIMPLE;
GO

BEGIN TRY
    BEGIN TRANSACTION
        -- Bulk logged 
        SELECT  *
        INTO    dbo.bigtable_intermediate
        FROM    dbo.bigtable
        WHERE   Id % 2 = 0;

        -- minimal logged because DDL-Operation 
        TRUNCATE TABLE dbo.bigtable;  

        -- Bulk logged because target table is exclusivly locked! 
        SET IDENTITY_INSERT dbo.bigTable ON;
        INSERT INTO dbo.bigtable WITH (TABLOCK) (Id, c1, c2, c3)
        SELECT Id, c1, c2, c3 FROM dbo.bigtable_intermediate ORDER BY Id;
        SET IDENTITY_INSERT dbo.bigtable OFF;
    COMMIT
END TRY
BEGIN CATCH
    IF @@TRANCOUNT > 0
        ROLLBACK
END CATCH

ALTER DATABASE DeleteRecord SET RECOVERY FULL;
GO

Update November 2016

If you plan on storing this much data in a single table: don't. I highly recommend you consider table partitioning (either manually or with the built-in features if you're running Enterprise edition). This makes dropping old data as easy as truncating a table once a (week/month/etc.). If you don't have Enterprise (which we don't), you can simply write a script which runs once a month, drops tables older than 2 years, creates next month's table, and regenerates a dynamic view that joins all of the partition tables together for easy querying. Obviously "once a month" and "older than 2 years" should be defined by you based on what makes sense for your use-case. Deleting directly from a table with tens of billions of rows of data will a) take a HUGE amount of time and b) fill up the transaction log hundreds or thousands of times over.

4
  • 15
    Up to 10.5 billion, still chugging. Just don't try to execute COUNT(). ;) Jan 12, 2015 at 14:15
  • 6
    It's been a year, we're at 16.5 billion rows. We just added an additional data source, so it's growing a bit faster now. We've also moved this database to its own SQL instance to allow us to dedicate memory without starving the other databases on the server. I am still able to chart any data point over any 24-hour period in the last 3 years in less than a second. Our analysts love it. Feb 12, 2016 at 20:28
  • i know it's been a while, but can you tell me on what kind of hardware you were running this database? Very curious since we have a table of 5 billion rows, growing 1 billion a year, and ik would like to find out if this is starting to get problematic in the future
    – Jeroen1984
    Dec 8, 2016 at 11:29
  • 4
    @Jeroen1984 It's a virtual machine running on a Hyper-V host ProLiant DL360e Gen8 with two Intel(R) Xeon(R) CPU E5-2430 processors. The VM has 38GB of statically allocated RAM, and some number of virtual processors that I don't recall. Dec 8, 2016 at 18:47
37

It's hard to give a generic answer to this. It really depends on number of factors:

  • what size your row is
  • what kind of data you store (strings, blobs, numbers)
  • what do you do with your data (just keep it as archive, query it regularly)
  • do you have indexes on your table - how many
  • what's your server specs

etc.

As answered elsewhere here, 100,000 a day and thus per table is overkill - I'd suggest monthly or weekly perhaps even quarterly. The more tables you have the bigger maintenance/query nightmare it will become.

1
  • 14
    I'd like to re-enforce "bigger maintenance/query nightmare" - from personal experience I'd avoid splitting into tables like the plague. Dec 5, 2011 at 9:24
22

I do not know of a row limit, but I know tables with more than 170 million rows. You may speed it up using partitioned tables (2005+) or views that connect multiple tables.

0
20

I don't know MSSQL specifically, but 36 million rows is not large to an enterprise database - working with mainframe databases, 100,000 rows sounds like a configuration table to me :-).

While I'm not a big fan of some of Microsoft's software, this isn't Access we're talking about here: I assume they can handle pretty substantial database sizes with their enterprise DBMS.

I suspect days may have been too fine a resolution to divide it up, if indeed it needs dividing at all.

6

We have tables in SQL Server 2005 and 2008 with over 1 Billion rows in it (30 million added daily). I can't imagine going down the rats nest of splitting that out into a new table each day.

Much cheaper to add the appropriate disk space (which you need anyway) and RAM.

5

It depends, but I would say it is better to keep everything in one table for that sake of simplicity.

100,000 rows a day is not really that much of an enormous amount. (Depending on your server hardware). I have personally seen MSSQL handle up to 100M rows in a single table without any problems. As long as your keep your indexes in order it should be all good. The key is to have heaps of memory so that indexes don't have to be swapped out to disk.

On the other hand, it depends on how you are using the data, if you need to make lots of query's, and its unlikely data will be needed that spans multiple days (so you won't need to join the tables) it will be faster to separate out it out into multiple tables. This is often used in applications such as industrial process control where you might be reading the value on say 50,000 instruments every 10 seconds. In this case speed is extremely important, but simplicity is not.

3

We overflowed an integer primary key once (which is ~2.4 billion rows) on a table. If there's a row limit, you're not likely to ever hit it at a mere 36 million rows per year.

3

You can populate the table until you have not enough disk space.

For better performance you can try migration to SQL Server 2005 and then partition the table and put parts on different disks (if you have RAID configuration that could really help you). Partitioning is possible only in enterprise version of SQL Server 2005. You can look at the partitioning example at this link.

Also you can try to create views for most used data portion, that is also one of the solutions.

0

Largest table I've encountered on SQL Server 8 on Windows2003 was 799 million with 5 columns. But whether or not it's good will is to be measured against the SLA and usage case - e.g. load 50-100,000,000 records and see if it still works.

1
  • 2
    Not sure this is really an answer at all. Sep 28, 2012 at 6:25
-1
SELECT Top 1 sysobjects.[name], max(sysindexes.[rows]) AS TableRows, 
  CAST( 
    CASE max(sysindexes.[rows]) 
      WHEN 0 THEN -0 
      ELSE LOG10(max(sysindexes.[rows])) 
    END 
    AS NUMERIC(5,2)) 
  AS L10_TableRows 
FROM sysindexes INNER JOIN sysobjects ON sysindexes.[id] = sysobjects.[id] 
WHERE sysobjects.xtype = 'U' 
GROUP BY sysobjects.[name] 
ORDER BY max(rows) DESC
1
  • I ran this query and got this result. I have UrlCategories table in my database. So what does this result means? Name TableRows L10_TableRows UrlCategories 7 0.85 Feb 22, 2013 at 16:09
-5

Partition the table monthly.That is the best way to handle tables with large daily influx ,be it oracle or MSSQL.

1
  • 4
    Not sure how this is an answer to the specific question asked. Sep 26, 2012 at 7:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.