3

I know there's many of the same subjected question here in SO, my question is if I want to delete say around 1K rows rather few, given a List<int> of RecordID, I could avoid using a DataTable, and send the list translated into a statement:

string ParmRecordsToDelete_CsvWhereIN = "("
for(int CurIdx=0; CurIdx < RecIdsToDelete.Count; CurIdx++)
{
  ParmRecordsToDelete_CsvWhereIN += RecIdsToDelete[CurIdx] + ", ";
  //this method to create passed parameter 
  //logic to remove on last Coma on last Index..

  //or use stringJoin and somehow remove the last coma 
}
ParRecordsToDelete_CsvWhereIN +=")";

This will create something Like "('1','2','3'......)"

and then create a SqlCommand to call a stored procedure:

Delete * From @TblName WHERE @ColName IN @RecordsToDeleteCsvWhereIN

Is this an efficient approach? Is there a limit to the length of single parameter? I guess it's the N/VARCHAR(MAX) length.

I guess if it's not a kinda hackish solution it would not be limited with length...

What would be the best fast solution, or am I on the right track as it is?

5
  • How do you obtain the Id's to delete? Are they user related (The user choose these items to delete)? If not, can you find them directly inside the procedure with? Also, what are you using to access data? ADO.NET? Entity Framework?
    – jpgrassi
    Dec 18, 2015 at 23:26
  • this can be done create a function on the database that accepts a List here is an example of what I use currently with a Select statement but you will have to mess around as a good learning experience use something like this Declare @SomeList varchar(8000) SET @SiteList = '1,2,3,4,5,6,7' SELECT * from tblSomeTable WHERE Some_KeyID IN (SELECT Item FROM fnParseInt(@SomeList,','))
    – MethodMan
    Dec 18, 2015 at 23:35
  • also you only need to say Delete FROM TABLE not Delete * FROM I would suggest you start reading some basic Sql Tutorials
    – MethodMan
    Dec 18, 2015 at 23:36
  • @jpgrassi i read a table with all content+columns, decide what records to delete based on File exists, if false add id to list. take list , delete by ids. no EnTT FW
    – Robb_2015
    Dec 18, 2015 at 23:47
  • @MethodMan i am not that familiar with fn i am more of SpMan so far can you please give a broader sample code on the list fn ?
    – Robb_2015
    Dec 18, 2015 at 23:50

3 Answers 3

10

You could use table valued parameters to hand this. The application layer would look something like

C#

var tvp = new DataTable();
tvp.Columns.Add("Id", typeof(int));

foreach(var id in RecIdsToDelete)
    tvp.Rows.Add(new {id});

var connection = new SqlConnection("your connection string");

var delete = new SqlCommand("your stored procedure name", connection)
{
  CommandType = CommandType.StoredProcedure
};

delete
  .Parameters
  .AddWithValue("@ids", tvp)
  .SqlDbType = SqlDbType.Structured;

delete.ExecuteNonQuery();

SQL

IF NOT EXISTS(SELECT * FROM sys.table_types WHERE name = 'IDList')
BEGIN
    CREATE TYPE IDList AS TABLE(ID INTEGER)
END


CREATE PROCEDURE School.GroupStudentDelete
(                                         
        @IDS IDLIST READONLY      
)                                         
AS

SET NOCOUNT ON;

BEGIN TRY
        BEGIN TRANSACTION

        DECLARE @Results TABLE(id INTEGER)

        DELETE 
        FROM TblName 
        WHERE Id IN (SELECT ID FROM @IDS)        

        COMMIT TRANSACTION
END TRY
BEGIN CATCH
        PRINT ERROR_MESSAGE();

        ROLLBACK TRANSACTION
        THROW; -- Rethrow exception
END CATCH
GO

There are a number of advantages to this approach over building strings

  • You avoid creating queries in the application layer, creating a seperation of concerns
  • You can more easily test execution plans and optimize queries
  • You are less vulnerable to SQL injection attacks, since your given approach would not be able to use a paramaterized query to build the IN clause
  • The code is more readable and illustrative
  • You do not end up building excessively long strings

Performance

There are some considerations about the performance of TVPs on large datasets.

Because TVPs are variables, they do not compile statistics. This means the query optimizer can fudge the execution plan sometimes. If this happens there a couple options :

  • set OPTION (RECOMPILE) on any TVP statements where indexing is an issue
  • write the TVP into a local temp and setup the indexing there

Here is a great article on TVP's with a good section on performance considerations, and what to expect when.

So if you are worried about hitting limits on string parameters, the table valued parameters might be the way to go. But in the end, it is hard to say without knowing more about the data set you are working with.

2
  • thank you for your well described answer , i know that link from yesterday, the link as i was searching my question, and today by reading it again the part of "Passing Table-Valued Parameters" about the types matching TVP which with google lead me to a custom implementation of your own class if you bother to,[ developmentsimplyput.blogspot.co.il/2012/10/… ] so now i will Test/benchMark both. i know TVP from couple of weeks ago, and when there's no better choice to be used, i pass a DataTable, in that link there's other custom Type..
    – Robb_2015
    Dec 19, 2015 at 9:45
  • How is DECLARE @Results TABLE(id INTEGER) used? What is that for? I've got the delete working in my project, but getting -1 returned from ExecuteNonQuery() Mar 24, 2020 at 19:42
4

IN @Parameter is not an option, such thing does not work. You can hard wire the ids into IN (1,2,3,4...), but that is bad. I measured some lightweight select of 30000 ids:

TVP: 339ms first run, 319ms second run
hard wired: 67728ms first run, 42ms second run

As You see, when SQL server has to parse enormous string, it takes really long. On second run the query plan can be taken from execution plan cache, unfortunately with big id ranges this is extremely unlikely. It just wastes execution plan cache.

TVP can scale into milion of ids without problem, hard wired string causes sql server to fail the query for less than 100000 ids. It has nothing to do with string max length, it just can not process it.

Btw. if You build string like this, use StringBuilder or string.Join, appending to string in loop is very inefficient.

1
  • thanks for your answer and details you specified on the performamce i was busy implementing a custom : ICollection<T> with , IEnumerator<T> with an internal T[] as the internal list 'where T is SqlDataRecord' all that instead of a heavy DataTable and all the effort only to later see @James remark on the roundtrips .. i wonder if the DataTable is really passed as an object and not row by row under the hood too...
    – Robb_2015
    Dec 20, 2015 at 1:58
4

A better approach would be to use a table-valued parameter. You have to define a type for the parameter, but it's more efficient than specifying values in a string, especially numeric or date values because the server doesn't have to parse the string to get out the individual values.

I'm not sure what your ID type is, but if it's 'BIGINT', for example:

IF NOT EXISTS (SELECT * FROM dbo.systypes WHERE name='IDList')
    CREATE TYPE IDList AS TABLE (Id BIGINT);
GO

To initialize the type, then to create a stored procedure using it, something like this:

IF NOT EXISTS (SELECT * FROM dbo.sysobjects WHERE name='DeleteMultipleRecords')
   EXECUTE sp_executesql N'CREATE PROCEDURE DeleteMultipleRecords AS BEGIN SET NOCOUNT ON; END'
GO
ALTER PROCEDURE [dbo].[DeleteMultipleRecords]
   @IDs IDList READONLY
AS
BEGIN
   SET NOCOUNT ON

    DELETE FROM [Table] WHERE Id IN (SELECT Id FROM @IDs)
END

You can also use it with dynamic SQL from C#.

3
  • the type is Int32 in that specific case, i have used TVP it is greate with Write To Server on a bunch of records rather a few or one list of columns and to pass it as a parameter i would try to use (Google) "a Custom-written classes that implement IEnumerable<SqlDataRecord> and IEnumerator<SqlDataRecord>" instead of DataTable in that case
    – Robb_2015
    Dec 19, 2015 at 9:56
  • @Robb_2015: There would probably be no point in writing a custom class in this case. You want to break up the deletes into manageable chunks for two reasons, 1. To reduce locking the table for too long, 2. To reduce the bytes sent back and forth from the server, and 3. To reduce round trips between the client and server. A custom enumerator could only reduce memory usage on the client, but at the expense of #3, which is likely more important.
    – James
    Dec 19, 2015 at 15:31
  • i thought about #3 but thought round trips occurs in both or non of the two ... so i won't mind mem usage as much as overhead of round-trips, i wonder when its better to use custom class if so, if for every record there's a separated action against server
    – Robb_2015
    Dec 19, 2015 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.