I have a local SQL Server 2017 database, and I need to copy two tables to an Azure SQL Server database; one table has over 100 million rows of data including a "geography" type column. How do I do that?
I am right now running a bulk copy:
using (SqlConnection streamsConnection = new SqlConnection(streamsConnectionString))
{
streamsConnection.Open();
using (SqlConnection cloudConnection = new SqlConnection(cloudConnectionString))
{
cloudConnection.Open();
using (SqlCommand cmd = streamsConnection.CreateCommand())
using (SqlBulkCopy bcp = new SqlBulkCopy(cloudConnection))
{
bcp.DestinationTableName = "GroundDataNodes";
bcp.BatchSize = 200000;
bcp.BulkCopyTimeout = 1200;
bcp.NotifyAfter = 100000;
bcp.SqlRowsCopied += new SqlRowsCopiedEventHandler(s_SqlRowsCopied);
cmd.CommandText = "SELECT [Id],[nodeid],[latlon],[type],[label],[code],[lat],[lon]FROM[dbo].[GroundDataNodes]";
using (SqlDataReader reader = cmd.ExecuteReader())
{
bcp.WriteToServer(reader);
}
Console.WriteLine("Finished!");
Console.ReadLine();
}
}
}
But I'm quite new to the bulk load side and am wondering how I can improve this so it doesn't take weeks to run...
WHERE ID BETWEEEN @StartID AND @EndID
. Add awhile
loop around the outermostusing
such that no more than 100K rows are selected at a time. This will avoid performing the entire operation in a single large transaction.