I've tried to find solution for this problem twice before, but unfortunately those answers haven't provided permanent fix, so here I am, giving it another try.
I have an SQL Server stored procedure that returns list of 1.5 million integer IDs. I am calling this SP from ASP.NET/VB.NET code and executing a SqlDataReader:
m_dbSel.CommandType = CommandType.StoredProcedure
m_dbSel.CommandText = CstSearch.SQL.SP_RS_SEARCH_EX
oResult = m_dbSel.ExecuteReader(CommandBehavior.CloseConnection)
Then I am passing that reader to a class constructor to build Generic List(Of Integer). The code is very basic:
Public Sub New(i_oDataReader As Data.SqlClient.SqlDataReader)
m_aFullIDList = New Generic.List(Of Integer)
While i_oDataReader.Read
m_aFullIDList.Add(i_oDataReader.GetInt32(0))
End While
m_iTotalNumberOfRecords = m_aFullIDList.Count
End Sub
The problem is - this doesn't read all 1.5 million of records, the number is inconsistent, final count could be 500K or 1 million etc. (Most often "magic" number of 524289 records is returned). I've tried using CommandBehavior.SequentialAccess
setting when executing command, but the results turned out to be inconsistent as well.
When I am running SP in SSMS, it returns certain number of records almost right away and displays them, but then continues to run for a few seconds more until all 1.5 million records are done - does it have anything to do with this?
UPDATE
After a while I found that on very-very rare occasions the loop code above does throw an exception:
System.NullReferenceException: Object reference not set to an instance of an object. at System.Data.SqlClient.SqlDataReader.ReadColumnHeader(Int32 i)
So some internal glitch does happen. Also it looks like if I replace
While i_oDataReader.Read
m_aFullIDList.Add(i_oDataReader.GetInt32(0))
End While
that deals in Integers with
While i_oDataReader.Read
m_aFullIDList.Add(i_oDataReader(0))
End While
that deals in Objects - the code seems to run without a glitch and returns all records.
Go figure.
524288
is an exact power of 2 (2^19
). Not sure how/if that is relevant.m_aFullIDList.Add(i_oDataReader.GetInt32(0))
and just increment a counter variable do you still see the issue?List(of Integer)
may be trying to do a dynamic reallocation and failing. IIRC, these reallocations tend to happen around certain powers of two.