0

I have a local SQL Server 2017 database, and I need to copy two tables to an Azure SQL Server database; one table has over 100 million rows of data including a "geography" type column. How do I do that?

I am right now running a bulk copy:

using (SqlConnection streamsConnection = new SqlConnection(streamsConnectionString))
{
    streamsConnection.Open();

    using (SqlConnection cloudConnection = new SqlConnection(cloudConnectionString))
    {
        cloudConnection.Open();

        using (SqlCommand cmd = streamsConnection.CreateCommand())
        using (SqlBulkCopy bcp = new SqlBulkCopy(cloudConnection))
        {
            bcp.DestinationTableName = "GroundDataNodes";
            bcp.BatchSize = 200000;
            bcp.BulkCopyTimeout = 1200;
            bcp.NotifyAfter = 100000;
            bcp.SqlRowsCopied += new SqlRowsCopiedEventHandler(s_SqlRowsCopied);

            cmd.CommandText = "SELECT [Id],[nodeid],[latlon],[type],[label],[code],[lat],[lon]FROM[dbo].[GroundDataNodes]";

            using (SqlDataReader reader = cmd.ExecuteReader())
            {
                bcp.WriteToServer(reader);
            }

            Console.WriteLine("Finished!");
            Console.ReadLine();
        }
    }
}

But I'm quite new to the bulk load side and am wondering how I can improve this so it doesn't take weeks to run...

1
  • 1
    I suggest you perform the bulk insert in batches by ranges of the source table clustered index key. For example, WHERE ID BETWEEEN @StartID AND @EndID. Add a while loop around the outermost using such that no more than 100K rows are selected at a time. This will avoid performing the entire operation in a single large transaction.
    – Dan Guzman
    Aug 4, 2018 at 15:14

2 Answers 2

0

Give Azure Database Migration Services a Try. I Migrated an on-premise SQL Server with 10 Million Rows in a Days Time. But of course it also largely depends on your bandwidth.

Also: Use the multi CPU General Purpose Pricing Tier when you create your service instance to allow the service to take advantage of multiple vCPUs for parallelization and faster data transfer.

Temporarily scale up your Azure SQL Database target instance to the Premium tier SKU during the data migration operation to minimize Azure SQL Database throttling that may impact data transfer activities when using lower-level SKUs.

1
  • thanks for that. tried it and it does work for me but its just too slow. great tip though as its helped me for some smaller stuff Aug 6, 2018 at 15:29
0

so I have tried all sorts of upload stuff like the bulk upload, migration, backup etc but all had the same problem and that was my upload speed isn't up to it. It would work but take days to run. So I decided to write a server side bit of code to just populate it from there directly onto the database therefore taking my upload speeds out of the equation. I imagine if I had better upload speeds the migration tool would have worked fine even with the Geographic field etc. not quick but it would work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.