2

I've tried to find solution for this problem twice before, but unfortunately those answers haven't provided permanent fix, so here I am, giving it another try.

I have an SQL Server stored procedure that returns list of 1.5 million integer IDs. I am calling this SP from ASP.NET/VB.NET code and executing a SqlDataReader:

m_dbSel.CommandType = CommandType.StoredProcedure
m_dbSel.CommandText = CstSearch.SQL.SP_RS_SEARCH_EX
oResult = m_dbSel.ExecuteReader(CommandBehavior.CloseConnection)

Then I am passing that reader to a class constructor to build Generic List(Of Integer). The code is very basic:

Public Sub New(i_oDataReader As Data.SqlClient.SqlDataReader)

    m_aFullIDList = New Generic.List(Of Integer)

    While i_oDataReader.Read
        m_aFullIDList.Add(i_oDataReader.GetInt32(0))
    End While

    m_iTotalNumberOfRecords = m_aFullIDList.Count

End Sub

The problem is - this doesn't read all 1.5 million of records, the number is inconsistent, final count could be 500K or 1 million etc. (Most often "magic" number of 524289 records is returned). I've tried using CommandBehavior.SequentialAccess setting when executing command, but the results turned out to be inconsistent as well.

When I am running SP in SSMS, it returns certain number of records almost right away and displays them, but then continues to run for a few seconds more until all 1.5 million records are done - does it have anything to do with this?

UPDATE


After a while I found that on very-very rare occasions the loop code above does throw an exception:

System.NullReferenceException: Object reference not set to an instance of an object. at System.Data.SqlClient.SqlDataReader.ReadColumnHeader(Int32 i)

So some internal glitch does happen. Also it looks like if I replace

While i_oDataReader.Read
  m_aFullIDList.Add(i_oDataReader.GetInt32(0))
End While

that deals in Integers with

While i_oDataReader.Read
   m_aFullIDList.Add(i_oDataReader(0))
End While

that deals in Objects - the code seems to run without a glitch and returns all records.

Go figure.

34
  • 1
    If it is a stored procedure, please post the definition of the procedure. Aug 27, 2013 at 15:04
  • 1
    Can you put a simpler Select in the stored procedure.....and see if you get predictable results? Something like : select top 1500000000 so1.id from master.sys.sysobjects so1 cross join master.sys.sysobjects so2 cross join master.sys.sysobjects so3 /* select COUNT_BIG(*) from master.sys.sysobjects so1 cross join master.sys.sysobjects so2 cross join master.sys.sysobjects so3 */ Aug 27, 2013 at 17:17
  • 3
    524288 is an exact power of 2 (2^19). Not sure how/if that is relevant. Aug 27, 2013 at 17:32
  • 2
    @YuriyGalanter - Absolutely none whatsoever. If you remove the m_aFullIDList.Add(i_oDataReader.GetInt32(0)) and just increment a counter variable do you still see the issue? Aug 27, 2013 at 17:37
  • 1
    I like where @MartinSmith is going with this: your List(of Integer) may be trying to do a dynamic reallocation and failing. IIRC, these reallocations tend to happen around certain powers of two. Aug 27, 2013 at 17:48

2 Answers 2

4

Basically, as we've flogged out in the comments(*), the problem isn't with SqlDataRead, the stored procedure, or SQL at al. Rather, your List.Add is failing because it cannot allocate the additional memory for 2^(n+1) items to extend the List and copy your existing 2^n items into. Most of the time your n=19 (so 524289 items), but sometimes it could be higher.

There are three basic things that you could do about this:

  1. Pre-Allocate: As you've discovered, by pre-allocating you should be able to gwet anywhere from 1.5 to 3 times as many items. This works best if you know ahead of time how many items you'll have, so I'd recommend either excuting a SELECT COUNT(*).. ahead of time, or adding a COUNT(*) OVER(PARTITION BY 1) column and picking it out of the first row returned to pre-allocate the List. The problem with this approach is that you're still pretty close to your limit and could easily run out of memory in the near future...

  2. Re-Configure: Right now you are only getting at most 2^22 bytes of memory for this, when in theory you shoud be able to get around 2^29-2^30. That means that something on your machine is preventing you from extending your writeable Virtual Memory limit that high. Likely causes include the size of your pagefile and competition from other processes (but there are other possibilities). Fix that and you should have more than enough headroom for this.

  3. Streaming: Do you really need all 1.5 million items in memory at the same time? If not and you can determine which you don't need (or extract the info that you do need) on the fly, then you can solve this problem the same way that SqlDataReader does, with streaming. Just read a row, use it, then lose it and go on to the next row.

Hopefully this helps.

(* -- Thanks, obviously, to @granadaCoder and @MartinSmith)


If you really think that the problem rests solely with the List data structure (and not that you are just running out of memory), then there are some other ways to work around the List structure's allocation behavior. One way would be to implement an alternative List class (as IList(of Integer)).

Through the interface it would appear the same as List but internally it would have a different allocation scheme, by storing the data in a nested List(of List(of Integer)). Every 1000 items, it would create a new List(of Integer), add it to the parent nested list and then use it to add in the next 1000 items.

The reason that I didn't suggest this before is because, like pre-allocation, this may allow you to get closer to your memory limit, but, if that's the problem, you are still going to run out eventually (just as with pre-allocating) because this limit is too close to the actual number of items that you need (1.5 million).

15
  • Thanks for the detailed explanation. I do need the full list of IDs, it is used later on in paging mechanism where 50-100 IDs at a time a sent to DB to retrieve full set of data for those IDs only. I do like "Pre-allocate" solution, but I don't want to run an extra query to get record count (and I don't think there's another way to get records affected by Reader). If I preset it to the value I think would be max (e.g. 2000000) - does it allocate that memory right away? What happens if actual number is much lower? Aug 27, 2013 at 19:12
  • @YuriyGalanter If you pre-allocate then AFAIK it really does allocate that much. If you don't want the extra query, the how about the extra column? That way you can pick it out of the first row. Aug 27, 2013 at 19:15
  • I did found where it fails. List reallocates capacity by the power of 2 you were correct, e.g if it has reached 8 records - capacity switched to 16, when reached 16 - to 32 etc. When count reaches 524,288 (2^19) it switches capacity to 1,048,576 (2^20) - and that capacity change works fine, without any error, but immediately after that Reader.read return false and exit loop. Any idea how these could be related? I am still trying to find a middle ground between hard-coding the capacity in advance and returning extra data from query Aug 27, 2013 at 20:48
  • It's got to be something memory-related under the hood. But, I'd suggest taking it to Microsoft as it seems like a bug right now (SqlDataReaders behavior, that is). Aug 27, 2013 at 20:52
  • @YuriyGalanter I should mention that the Debugger could be interacting adversly with all of this also. Memory Allocation failure is such an insidious problem, it can bollix-up all kinds of seemingly unrelated things. Aug 27, 2013 at 20:56
0

Basically you read all record in SqlDataReader with select query I suggest you to add order by in your query and it sort all records in Acceding order and they also read in acceding order in SqlDataReader.

I also face this problem in my last project I have read more than 2 million records from database with unique id serialNo but this records are not come in sequence after 1000 records it jumps to 21, 00, 263th record and all records are come in wrong sequence.

Then I use (order by serialNo) this query and my problem is solved you not need to do anything extra only put order by in your select query and it will work for you

I hope this helps for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.