Questions tagged [sqlbulkcopy]
Lets you efficiently bulk load a SQL Server table with data from another source.
963
questions
1
vote
1
answer
72
views
DataTable and big SqlDecimal [closed]
I just found out that when you use DataTable and SqlBulkCopy in an attempt to load a SQL Server table, you can't use big decimal values.
DataTable only supports the .NET decimal type. .NET decimal is ...
2
votes
3
answers
71
views
Error when importing data from Excel to SQL Server with CLR
SQL DBA here, not a C# or a programmer guy. Below is some code I found online and tried to adapt to my needs:
public partial class StoredProcedures
{
[Microsoft.SqlServer.Server.SqlProcedure]
...
1
vote
1
answer
38
views
implement bulk copy in different environment
I have a problem about bulk copy command in PostgreSQL. For instance i have a sample command of copying file CSV to table like below
CREATE TEMPORARY TABLE AlterTable (
a1 varchar(50),
a2 ...
1
vote
1
answer
262
views
504 Gateway Timeout: STRANGE!!! while server accepting the file into DB the response is not making it to the Client
I have service that uploads and parses line by line into DataTable and records it via SQL bulk copy into the DB. Locally this service works fine with overall implementation in 20 secs and in LAN dev ...
0
votes
2
answers
60
views
WriteToServerAsync Not working as expected when Upgrade to .net 6
My current project was in .net 5 which I have upgraded to .net 6 version.
Earlier the code was working fine but after upgrade this code seems to be not working.
public async Task AddBulkAsync(...
0
votes
0
answers
154
views
SqlBulkCopy WriteToServer using a DataTable just does nothing
I'm trying to use SqlBulkCopy for inserting bigger amounts of data fast. For that I populate a DataTable with the data I want to insert (up to about 100,000 rows) and call the WriteToServer method. ...
0
votes
0
answers
46
views
Get primary keys after inserting data into SQL Server database using SqlBulkCopy [duplicate]
I'm using SqlBulkCopy to insert records into a SQL Server database. I have a requirement to save the data into 3 tables based on the primary key which I get after inserting into 1st table. As the ...
0
votes
0
answers
110
views
Why would SqlBulkCopy be extremely slow sometimes?
I'm trying to diagnose a performance issue with SqlBulkCopy.
I have a point in my code where most of the time, one row is being inserted, but sometimes hundreds of rows can be inserted, most of the ...
0
votes
0
answers
68
views
Need to improve speed of processing
"I have a script of insert and update in a folder which is executed by window service. In the window service, there is a function that will check and get the batch of 10 files and executes and ...
0
votes
1
answer
447
views
BULK INSERT into Managed Instance from a file in Azure Files?
How can I use BULK INSERT to load files that are in Azure Files, from a Managed Instance? Everything I can find talks about Azure Blob storage, but I've got an old Perl script that we don't want to ...
0
votes
1
answer
267
views
Powershell, SQL Server, SQLBULKCOPY: Syntax to specify codepage or encoding
I'm using a Powershell script found at ((https://blog.netnerds.net/2015/01/powershell-high-performance-techniques-for-importing-csv-to-sql-server/) to load data from quotes delimited, comma separated ....
0
votes
1
answer
200
views
SqlBulkCopy failing when trying to handle more than 1 row
I'm trying to perform a bulk upload operation to a SQL Server database. I'm working in .NET 7, and I'm using a SqlBulkCopy instance to upload my data as a DataTable.
I've had all sorts of issues ...
1
vote
1
answer
186
views
How to improve SqlBulkCopy performance with Azure SQL
When I run a program on local DB to insert 100,000 records. It take around 2 seconds to import. After switch to Azure SQL the time increase with high percentage of DTU.
This is my code
using (var ...
1
vote
1
answer
496
views
Copy contents of the Excel file from a stream asynchronously using SqlBulkCopy
I have a project where I need to copy the contents of the .xlsx file I received in Web API Controller (in the form of the Stream from MultipartReader) to SQL Server Database. I'm using SqlBulkCopy for ...
1
vote
1
answer
223
views
Powershell, SQL Server, SQLBULKCOPY: Importing NULL Values in .csv
I'm using a Powershell script found at ((https://blog.netnerds.net/2015/01/powershell-high-performance-techniques-for-importing-csv-to-sql-server/) to load data from quotes delimited, comma separated ....
2
votes
1
answer
277
views
C# Using foreach loop to bulkcopy data to SQL Server
I have a C# console app which reads million source data from a CSV file and inserts them into SQL Server in batches.
I group the data by 1000 count and use foreach to loop the groups. Each loop ...
1
vote
0
answers
68
views
SQL Bulk Copy by mapping columns dynamically does not work but works manually
I am trying to do a columnMappings for a bulk insert but i cant get the Columnmapping to work in a for loop but when i do it manually it works just fine.
I get this error when trying a dynamic ...
0
votes
1
answer
134
views
Cannot remove columns of DataTable before using SqlBulkCopy
I want to remove columns of DataTable that have name start with "##" then bulk to database using C# and SQL Server.
This is my code:
private bool WriteData(DataTable _dt)
{
DataTable dt =...
0
votes
1
answer
371
views
SqlBulkCopy.WriteToServerAsync() does not write to target SQL Server table, bulkCopy.WriteToServer() does
Just as the title states. I am trying to load a ~8.45GB csv file with ~330 columns (~7.5 million rows) into a SQL Server instance, but I'm doing the parsing internally as it the file has some quirks ...
0
votes
0
answers
203
views
SqlBulkCopy from Oracle to SQL Server with DataReader is slow (100k to 20M)
I have been working on sending a table from Oracle to SQL Server with SqlBulkCopy.
I am doing it by reading with a datareader from oracle and then streaming into SQL Server. I have tried with ...
2
votes
0
answers
81
views
Timeout exception and 'Cannot access destination table #name' on bulk.WriteToServer(readerTabel) with small timeout /SQL wait time/block
I have a problem with SqlBulkCopy that sometimes(pretty often) i get a timeout error and an sql error 'Cannot access destination table #temp'.
My timeout on bulkcopy is 5 seconds. After further ...
0
votes
0
answers
100
views
c# Save JSON response to Database table
Ok I tried this for a couple of days now.
I'm trying to read JSON response and insert/update it to DB table (already exists) based on a condition from the response. I have created a Console app.
What ...
0
votes
0
answers
65
views
{"Failed to convert parameter value from a Object[] to a Decimal."} in SqlBulkCopy using SqlParameter [duplicate]
I need to insert bulk data into a SQL Server database.
The data is fetched in SqlParameter.
foreach (DataColumn c in data.Columns)
{
SqlDbType dbType = GetSqlDbTypeFromDotnetType(c.DataType);
...
1
vote
1
answer
124
views
How to get acceptable performance saving data tables to database using table adapter Update (update in batches)?
I am having some performance issues updating a large datatable to SQL server using the table adapter functionality.
I am doing exactly what this examples is doing:
https://learn.microsoft.com/en-us/...
2
votes
1
answer
392
views
How to get newly inserted Ids from identity primary key column when using SqlBulkCopy
I'm using C# and .NET Core 6. I want to bulk insert about 100 rows at once into database and get back their Ids from BigInt identity column.
I have tried lot of different variants, but still do not ...
0
votes
1
answer
108
views
SSIS C# SQLBulkCopy .csv file Error: Failed to convert parameter value from a String to a Boolean. String not recognized as a valid Boolean
I am trying to create a generic module that will load .csv files into SQL tables. The SQL tables are already created and their names, and the name of the file, will be passed as parameters. This what ...
0
votes
1
answer
85
views
Load of generic .csv files into SQL Table using BulkCopy
I am trying to create a generic module that will load .csv files into SQL tables. The SQL tables are already created and their names, and the name of the file, will be passed as parameters. This ...
0
votes
1
answer
259
views
Using Powershell to copy large Oracle table to SQL Server - memory issue
We are trying to copy data from a large Oracle table (about 50M rows) to SQL Server, using Powershell and SQLBulkCopy. The issue with this particular Oracle table is that it contains a CLOB field and ...
0
votes
0
answers
176
views
SqlBulkCopy Slow inserting with async (from Oracle to SQL Server)
I'm trying to do an insert into SQL Server from Oracle using SqlBulkCopy, but it is too slow.
Here are some details:
SQL Server 2014 with recovery model "Simple" (clean database for this).
...
1
vote
1
answer
1k
views
SqlBulkCopy in C# for copy from Oracle view to SQL Server table too slow
I'm trying to use SqlBulkCopy to copy from Oracle to SQL Server, I have a view in oracle with 12 million records so I'm looking for a fast way to copy.
I've done some analysis and I would be able to ...
0
votes
1
answer
2k
views
How to use SqlBulkCopy.WriteToServer to Bulk Insert rows into SQL Server , ONLY if Not Exists
We are using SqlBulkCopy.WriteToServer to bulk insert to SQL Server and works very well. However, it fails when a record already exists. What we need is to "ignore" those rows that already ...
2
votes
0
answers
74
views
Is it possible to execute multiple "COPY FROM" commands from a file?
Via pgAdmin, I created a backup file with the following format:
COPY alpha (columns) FROM stdin;
data TAB data TAB ...
data TAB data TAB ...
data TAB data TAB ...
\.
COPY beta (columns) FROM stdin;
...
0
votes
0
answers
129
views
I am trying to save a DataTable to SQL Server using SqlBulkCopy() but I get an exception
Here is my code to save the DataTable to SQL Server:
SqlBulkCopy bcp = new SqlBulkCopy(con);
bcp.DestinationTableName = "Hawalas";
ListtoDataTableConverter converter = new ...
1
vote
1
answer
1k
views
Importing CSV to SQL Server using bulkcopy
I'm trying to import my CSV files into SQL Server. I found this code and it works perfectly and very fast:
# Database variables
$sqlserver = "servername"
$database = "datebasename"
...
1
vote
1
answer
188
views
Bulk insert from CSV - country codes are not coming to SQL table
I have a simple CSV file including 60 rows of data like below:
"Name","City","Region","Primary Phone","Postal Code"
"MyCompany Inc.","...
0
votes
0
answers
103
views
WriteToServer does not throw exception if destination table does not exist in database
I'm using SqlBulkCopy to bulk insert data in C#, but it's not throwing an exception if the destination table is not present in the database.
Here is my code:
SqlConnection connection = new ...
0
votes
0
answers
153
views
SQL bulk copy C#
I have an ASP.NET application to insert an excel file as a bulk I have an Excel file containing 50,000 records. I am using SQLBulkCopy to insert it. I can insert it successfully from the development ...
0
votes
1
answer
578
views
Error 2200: Copy Activity - Partitioned Parquet Column Not Found
Things I've tried:
I have checked the pipeline's JSON and none of the columns have a space in them.
I added in the "physicalType: "UTF8"" to the JSON as it was missing beforehand.
...
0
votes
1
answer
298
views
How can I make AllowDBNull work for GUID?
I am using DataColumn to create a new column for a DataTable. I am using AllowDBNull to specify that a column can have NULL values.
This works fine except for the case when I have a column with ...
0
votes
0
answers
193
views
SqlBulkCopy when column types are unknown
I am working on a C# app that imports csv data to SQL server. I am using CsvHelper to achieve this. The use case here is that the app will be used by anyone who wishes to import data from CSV to SQL ...
0
votes
0
answers
359
views
Export Data from Sql Server to To Excel Using Sqlbulkcopy
is any one here have the idea to export/copy data from sql server to excel.
I use OPENROWSET and its working but the client wants to get the data using bulk copy
is there a way to get the data using ...
0
votes
3
answers
900
views
SqlBulkCopy WriteToServer with an IDataReader instead of DataTable and Programmatically Adjusted Field Values
We have a working code in C# that utilizes SqlBulkCopy to insert records into a table from a stored procedure source. At a high-level:
Reads data from a stored procedure that puts the records into a ...
1
vote
1
answer
292
views
SQL Insert statements in C# on Azure DB - Running very slow
I'm working on an importer in our web application. With the code I currently have, when you are connecting via local SQL server, it runs fine and within reason. I'm also creating a .sql script that ...
1
vote
0
answers
1k
views
Using SQL Bulkcopy based on Db contexts connection
In my .NET Core project (.NET 5) I'm using EF Core 6 and for SQL auth, it's using Azure AD Identity authentication.
Here is my startup where DB context is initialized:
services.AddDbContext<...
-1
votes
1
answer
98
views
Prevent SqlBulkCopy to change XML value
I am using SqlBulkCopy to transfer data from one database to another. My problem is that my XML column will be changed during the bulk copy.
I think that the SqlBulkCopy class is parsing/computing the ...
2
votes
3
answers
820
views
Bulk Insert with NTS geometries in .NET Core
As you probably know, inserting data into a table the "Entity Framework"-way is incredibly slow when you try to insert a lot of rows. One other way is to use SqlBulkCopy which does a great ...
0
votes
2
answers
433
views
Uploading 100,000 rows into a SQL Server database
What's the fastest way to upload bulk (over 100,000 records) from an ASP.NET MVC application?
I tried using the following methods:
BulkCopy but I keep getting this error:
Message: Error writing to ...
2
votes
1
answer
217
views
SqlBulkCopy throws "Operation is not valid due to the current state of the object"
I'm attempting to create a custom DataReader for an IAsyncEnumerable<T> collection in order to load the records into a database table via SqlBulkCopy. I'm following along the lines of the code ...
5
votes
1
answer
11k
views
What is the difference between polybase and bulk insert, copy methods in azure data factory and when to use them?
Can some one please elaborate on when to use Polybase versus bulk insert in azure datafactory, what are the differences between these two copy methods?
2
votes
1
answer
312
views
BCP multiple files in/out using Azure Active Directory Interactive
I am trying to export/import tables using bcp, I am able to do it one by one as the documentation says using Azure Active Directory Interactive (I require Multifactor authentication to access the DB), ...