3

I'm not able to insert the bulk amount of data into Azure SQL server DB using C# webapi

Consider

I want to insert 60K> data in SQL. In my local sql server there is no problem but in Azure SQL its getting connection timed-out

My approach:(All are working in local sql server but not in Azure sql server)

1) Tried using EF its inserting record one by one (For 10000 approx. 10 min,mostly timeout)

2) Tried using Bulk insert Extension along with EF 3) Tried in SqlBulkCopy

4) Tried increasing connection time out in connection string

5) Tried increasing command time out in Dbcontext.

Exception StackTrace

Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlBulkCopy.RunParser(BulkCopySimpleResultSet bulkCopyHandler)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinuedOnSuccess(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
   at System.Data.SqlClient.SqlBulkCopy.WriteToServer(DataTable table, DataRowState rowState)

Is there any solution for it or any configuration to be changed in Azure?

Update

Code used for bulk insert

  using (var dbConnection = new DBModel().Database.Connection as SqlConnection)
                {
                    dbConnection?.Open();
                    using (var sqlBulkCopy = new SqlBulkCopy(dbConnection))
                    {
                        try
                        {
                            /* ColumnMapping
                             * Column is mapped to DB Column to DataTable Column
                             *
                             */
                            sqlBulkCopy.EnableStreaming = true;
                            sqlBulkCopy.BulkCopyTimeout = 500;
                            sqlBulkCopy.DestinationTableName = "LogTable";
                            //dt is object of the Datatable
                            sqlBulkCopy.WriteToServer(dt);
                        }
                        catch (Exception ex)
                        {

                        }
                    }


                }
12
  • what is your database tier ? Commented Jan 17, 2017 at 14:10
  • Can you post the code you have used when tried with SqlBulkCopy? You must at least modify default value for BulkCopyTimeout and BatchSize property. Commented Jan 17, 2017 at 15:18
  • @TheGameiswar Its Standard
    – Sreemat
    Commented Jan 18, 2017 at 5:25
  • @JonathanMagnan I tried that too its not working
    – Sreemat
    Commented Jan 18, 2017 at 5:25
  • 1
    @Sreemat, you forget to add a Batch Size. You are currently trying to insert 60,000k rows at once which is way too much probably for your current database tier. Start by trying to set a very low Batch Size (like 100), to see if that's working. Commented Jan 18, 2017 at 13:22

4 Answers 4

5

I'd suggest you set sqlBulkCopy.BatchSize to a reasonable amount, instead of inserting everything in one batch. Depending on the data you're inserting, try starting with 10.000 and work your way up or down until you're satisfied with the performance.

Edit for some extra clarification: When you consider your batch size, you need to take into consideration that SqlBulkCopy will need to not only insert the data, but also read AND send it - the last part is probably the reason why it works on your local SQL server, but not on Azure - it also means that, if you're working with a large dataset, that you will need to work with lower batch size, or a considerably higher BulkCopyTimeout setting, to allow each batch the chance to finish before reaching the timeout limit.

You can read more on batch sizes in this post. What is the recommended batch size for SqlBulkCopy?

Other option:
I was reading up on this, and it could simply be because your insert reaches a critical DTU (Database Transaction Unit, basically a measure of the servers combined resources) usage point.

Performance levels are calibrated and governed to provide the needed resources to run your database workload up to the max limits allowed for your selected service tier/performance level. If your workload is hitting the limits in one of CPU/Data IO/Log IO limits, you will continue to receive the resources at the maximum allowed level, but you are likely to see increased latencies for your queries. These limits will not result in any errors, but just a slowdown in your workload, unless the slowdown becomes so severe that queries start timing out.

Taken from this link: https://azure.microsoft.com/da-dk/blog/azure-sql-database-introduces-new-near-real-time-performance-metrics/
Try starting the copy again while monitoring the DTU usage and see if it's on 100% for long(er) periods. If that is the case, you might want to up your pricing tier scale for the database.

4
  • You can give this a try. galleryserverpro.com/…
    – lmms90
    Commented Jan 19, 2017 at 13:45
  • Yup @lmms90 me too got this link
    – Sreemat
    Commented Jan 19, 2017 at 13:54
  • @Sreemat - I expanded my answer with a second option, maybe it will be of interest
    – lmms90
    Commented Jan 24, 2017 at 12:57
  • My observations are a SqlBulkCopy with maxed out DTUs will make the sql server just give up with a timeout error. You can set the timeout to be next year sometime and it doesn't matter; you'll still get a timeout error, often at a very odd time ( like 28 seconds)...
    – user610217
    Commented Feb 4, 2019 at 13:04
1

I've been dealing with Bulk insert during last two days, here is a Generic Bulk Insert Class that allows you to exclude some columns and, in case of a DbGeography Property is found it cast it to SqlGeography:

 public class BulkInsert<T> where T : class
{
    #region Fields

    private readonly LoggingService _logger = new LoggingService(typeof(BulkInsert<T>));
    private string _connectionString;
    private string _tableName;
    private IEnumerable<string> _excludedPropertyNames;
    private int _batchSize;
    private IEnumerable<T> _data;
    private DataTable _dataTable;

    #endregion

    #region Constructor

    public BulkInsert(
        string connectionString,
        string tableName,
        IEnumerable<T> data,
        IEnumerable<string> excludedPropertyNames,
        int batchSize = 1000)
    {
        if (string.IsNullOrEmpty(connectionString)) throw new ArgumentNullException(nameof(connectionString));
        if (string.IsNullOrEmpty(tableName)) throw new ArgumentNullException(nameof(tableName));
        if (data == null) throw new ArgumentNullException(nameof(data));
        if (batchSize <= 0) throw new ArgumentOutOfRangeException(nameof(batchSize));

        _connectionString = connectionString;
        _tableName = tableName;
        _batchSize = batchSize;
        _data = data;
        _excludedPropertyNames = excludedPropertyNames == null ? new List<string>() : excludedPropertyNames;
        _dataTable = CreateCustomDataTable();
    }

    #endregion

    #region Public Methods

    public void Insert()
    {
        using (var connection = new SqlConnection(_connectionString))
        {
            connection.Open();
            SqlTransaction transaction = connection.BeginTransaction();

            using (var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.Default | SqlBulkCopyOptions.KeepIdentity, transaction))
            {
                bulkCopy.BatchSize = _batchSize;
                bulkCopy.DestinationTableName = _tableName;

                // Let's fix tons of mapping issues by
                // Setting the column mapping in SqlBulkCopy instance:
                foreach (DataColumn dataColumn in _dataTable.Columns)
                {
                    bulkCopy.ColumnMappings.Add(dataColumn.ColumnName, dataColumn.ColumnName);
                }

                try
                {
                    bulkCopy.WriteToServer(_dataTable);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex.Message);
                    transaction.Rollback();
                    connection.Close();
                }
            }

            transaction.Commit();
        }
    }

    #endregion

    #region Private Helper Methods

    private DataTable CreateCustomDataTable()
    {
        PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));
        var table = new DataTable();
        foreach (PropertyDescriptor prop in properties)
        {
            // Just include the not excluded columns
            if (_excludedPropertyNames.All(epn => epn != prop.Name))
            {                  
                if (prop.PropertyType.Name == "DbGeography")
                {
                    var type = typeof(SqlGeography);
                    table.Columns.Add(prop.Name, type);
                }
                else
                {
                    table.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
                }
            }
        }
        foreach (T item in _data)
        {
            DataRow row = table.NewRow();
            foreach (PropertyDescriptor prop in properties)
            {
                // Just include the values in not excluded properties 
                if (_excludedPropertyNames.All(epn => epn != prop.Name))
                {
                    if (prop.PropertyType.Name == "DbGeography")
                    {                           
                        row[prop.Name] = SqlGeography.Parse(((DbGeography)prop.GetValue(item)).AsText()).MakeValid();
                    }
                    else
                    {
                        row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
                    }
                }
            }
            table.Rows.Add(row);
        }
        return table;
    }

    #endregion

}

And well, It has to be used like:

var myEntityBulk = new BulkInsert<MyEntity>(
    _mYConnectionString,
     "MyEntities", 
      myEntities, 
      new[] { "ObjectState","NavigationPropertyOne", "NavigationPropertyTwo" }
);
myEntityBulk.Insert();

I hope it helps, I'm quite sure it will... Yesterday I bulked with it &M entities in few minutes.

0

Firstly, please make sure whether you could connect to Azure SQL database and do operation without timeout error if you just insert a record to database.

Secondly, please check whether you override the default timeout value by using the CommandTimeout property on the ObjectContext, and you could try to set 0 (that indicates no limit) for BulkCopyTimeout property.

Besides, please make sure whether it hit limits of your Azure SQL database while you do bulk insert, you could try to change the service tier and performance level of your database.

2
  • Firstly- Yes it connecting to Azure I'm able to fetch data and insert row less than 50K but if its more than 50K is getting timeout.Secondly - tired this option too not working
    – Sreemat
    Commented Jan 18, 2017 at 6:25
  • SqlBulkCopy has nothing to do with ObjectContext and Entity Framework. The OP should set a small batch size first eg 100 or 500. Increasing the timeout is another option, setting it to 0 though is not a very scaleable option - it will only freeze the application. The batch size can be increased to improve performance without causing timeouts Commented Jan 19, 2017 at 13:07
0

When using SqlBulkCopy, depend on the Route-Trip-Time (distance) between your web server and the Azure SQL server, there are two parameters you need to be aware of:

  1. BatchSize (for me I set batch size to 500)

  2. BulkCopyTimeOut: default is 30 seconds (I set to 60 secs)

    using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection)) {
            bulkCopy.BatchSize = array.Length;
            bulkCopy.DestinationTableName = tempTableName;
            bulkCopy.BulkCopyTimeout = 60;...}
    

With this configuration, my whole process of getting 20K+ events from EventStore to my Projection MicroService, then the microService Update/Write those 20K events to a Database took approximately 15 minutes (depends on the RTT- route trip time between the Event Store and the MicroService, and between Microservice and the database)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.