Questions tagged [bulkinsert]
Act of inserting multiple rows into a database simultaneously.
2,458
questions
2
votes
0
answers
49
views
PostgreSQL (ef core), bulk deleting / reinserting a table with 1.5 million rows gets slower each run
I have a small PostgreSQL table 1.5 million rows (uuid id, integer productId, integer rankType, text rankKey, integer position), no indexes except PK, every 10 minutes I need to empty/regenerate the ...
2
votes
1
answer
29
views
How to Bulk Import data from file into Netezza DB?
I want to bulk import data from a file present on my local into the Netezza Database using NZSQL language.
I tried with below query but, I am not able to do it.
create external table ext_tab (
...
0
votes
0
answers
18
views
Problem with bulk insert with mysql on nodeJs
I'm trying to do a bulk insert to a mysql table doing something like this:
const query = 'INSERT INTO player
(playerId, name, team, jersey, position, foot, dob)
...
0
votes
0
answers
26
views
Faster way to process 50 million records in Oracle
I'm trying to do a CDC on 60 million records in Oracle .
I need to check 60million source(TABLE_TEMP) records against the Target (TABLE) and do Insert , Update or soft Delete operations.
I tried to ...
0
votes
1
answer
49
views
Bulk Indexing Error in Elastic search Python
I am trying to ingest a simple Hello World example into a Data Stream using the Bulk call as shown in documemtation https://elasticsearch-py.readthedocs.io/en/v8.6.2/helpers.html#bulk-helpers
...
0
votes
0
answers
42
views
BULK INSERT with delimiter in column values
I am trying to insert the csv content in the table using BULK INSERT.
BULK INSERT temp.table
FROM test.csv
WITH
(FIRSTROW=2,
FIELDTERMINATOR='|',
ROWTERMINATOR='\n');
csv files is | separated, but ...
0
votes
0
answers
11
views
mysqlpp's (Tangensoft) insertfrom with ON DUPLICATE KEY UPDATE
I searched a lot in the docs, but can't find an example or explanation upon the (detailed!) usage of Query::InsertPolicy in conjunction to the mass insertion function "insertfrom()".
It just ...
0
votes
1
answer
25
views
Why I can't use more than one properties for UNIQUE constraint while loading the data using "bin/neo4j-admin database import full"?
I am trying to import the data using bin/neo4j-admin database import full command with 2 node files with header files and 1 relationship with header file.
But facing error that we can't use :ID for ...
0
votes
2
answers
55
views
Fast way to transfer table to a remote server as import/export does
I have a stored procedure with statements similar to this:
DELETE FROM [LinkedServer].[DB1].[dbo].[Table1]
DELETE FROM [LinkedServer].[DB1].[dbo].[Table2]
DELETE FROM [LinkedServer].[DB1].[dbo].[...
0
votes
0
answers
33
views
Alternate option for bulk data saveAll in spring jpa
if saveAll gets fail then all transaction get fail can there be any imlementation for the doing bulk database operation in spring jpa so that it save those records where there is no failure.
Process ...
0
votes
1
answer
100
views
Bulk Inserting CSV into Azure SQL with formatting file error
I'm trying to bulk insert data from a csv stored in a blob storage account in an Azure SQL database using a format file.
Here is what I am attempting to run:
BULK INSERT dbo.MyTable
FROM 'MyCSV.csv'
...
0
votes
1
answer
22
views
Insert multiple records in newly created column simultaneously in existing table in MySQL?
How to insert multiple values in a newly created column in existing table MySQL?
I am fairly new to SQL (beginner phase.)
I have an existing table named "members".
INSERT INTO members
(...
0
votes
1
answer
87
views
Filestream and using bulk insert via OPENROWSET gives File could not be opened. Operating system error code 50(The request is not supported.)
I felt in love with the filestream capacities of SQL Server giving the possibility to see only files and skip the complexity of hierarchies.
When I import a xml file in a table using openrowset I get ...
0
votes
0
answers
26
views
SQL Server - Bulk data Insert (talking data from a flat file tab delimited source) and creating temp tables
My goal is to take raw data I'm receiving from a file path and insert that into a new table or temporary table. The problem is there are no headers for the data I am getting. The file is a flat file ...
0
votes
0
answers
24
views
Can't see the inserted files in my database using Cursor
I have this code and executes with no issues but i dont see the tables or files that are supposed to be in the 'Names' Database here is the queries
And this is the data im trying to insert in bulk
...
3
votes
0
answers
84
views
Microsoft SQL Server Bulk Insert NOT failing when inserting bigint values into int column
We recently had an identifier column move from int values to bigint values. An ETL process which loads these values was not updated. That process is using SQL bulk insert, and we are seeing ...
0
votes
0
answers
175
views
How can I bulk insert records with UUID primary keys using insert_all in Rails?
I have a controller action which, upon creation of a parent object, generates and saves a batch of M-M associations. The database is Postgres and all relevant tables in the application use UUIDs.
When ...
0
votes
1
answer
65
views
Send large amount of data in Oracle in C#
[BEGINNER]
Sorry for my English.
I've been working for a new company for a short time and I consider myself a novice in BDD, since apart from school I haven't practiced since.
[ENVIRONMENT]
I am ...
0
votes
1
answer
18
views
Append Data to A DolphinDB Table with R
I have a DolphinDB table "t1". Now I want to append data.frame in R to "t1", how can I do? (Table schema cannot be taken into consideration. The number of columns of data.frame and ...
0
votes
1
answer
56
views
Sync data from remote api and save it to my localdb
I want to sync data from remote api! something like 1M record! but the whole process talks about 5Mins.
as a user experience, that's very bad thing to do! I want the whole process takes less than 1S!
...
0
votes
1
answer
103
views
CSV insertion using SQLCMD, how to do it?
I have been reading a lot of forums and tutorials where other users ask the same but the answer is almost the same always: use BCP.EXE (BULK INSERT) instead to import, and also export with BCP.EXE ...
1
vote
2
answers
216
views
How to read from text file and insert the data into PostgreSQL using Nodejs?
This text file contains more than 500k rows which I want to read and insert into PostgreSQL table using nodejs. These blanks spaces should also be read and saved like this empty space only.
I wrote ...
0
votes
1
answer
29
views
Pass table as argument to postgreSQL function?
is it possible to pass table/list/json as argument to postgreSQL function from .net core API? The requirement is like for an invoice, there are multiple line items. so we want to save invoice first ...
0
votes
0
answers
30
views
What does bulkCreate return?
I have searched everywhere and I cant get the answer to this. What does bulkCreate operation in sequelize (with postgreSQL) return? I'm getting weird response there so I want to know the exact kind of ...
0
votes
1
answer
68
views
bulk insert rowterminator not working properly
I am trying to import csv (1 million records) file to sql server table using bulk insert
Example of the csv data is as below:
"Phone Numbers","Opstype"
"9827259163","...
1
vote
1
answer
170
views
Bulk load data conversion error (type mismatch or invalid character for the specified codepage)
I am trying to use bulk insert to upload a txt file into a database I created.
Here is the script:
create table Producer(
ProducerID char(10) not null primary key,
Producer ...
0
votes
1
answer
44
views
I need to load a JSON file which has 100k records into Elasticsearch by using Python API. I was new to both the stack. How can i load it?
I need to create an index and load the data into Elasticsearch. How can i do it by using python? I need to do bulk data operations. I wanted to do it without CURL.
I was using 8+ version of ...
0
votes
1
answer
93
views
How to INSERT list mariadb where multiple values are excpected only list given
I want to insert a list (bulk insert) into mariadb using python. However the insert has 5 arguments and I only want to give the list as an argument. It will make sense if you read the code.
If I try ...
0
votes
0
answers
88
views
Bulk data insert into databricks
Currently, I am working on databricks that I have access to through a token. I can read data quite fast by pandas.read_sql_query(). However, it was prolonged when inserting data into tables and I used ...
1
vote
1
answer
62
views
Getting the desired view/result of XML in SQL Server (Bulk Insert XML into table)
I want to bulk-insert XML data into SQL Server tables.
What have I achieved so far?
I have successfully loaded the XML file into SQL Server and fetched XML data.
What's the problem Now?
The retrieved ...
0
votes
0
answers
113
views
sql alchemy bulk insert too slow
I am having performance issues with bulk insertion using sqlalchemy. I have these two database models:
@generic_repr
class CallLog(Base):
# NOTE: Until bullseye upgrade, this schema must match ...
0
votes
0
answers
81
views
How auto increment work for parallel bulk inserts
I have an application that processes files and transform them to bcp files that are imported into a SQL server database using bulk insert.
The application is using parallel threading, and as result ...
0
votes
1
answer
113
views
SqlBulkCopy.WriteToServerAsync() does not write to target SQL Server table, bulkCopy.WriteToServer() does
Just as the title states. I am trying to load a ~8.45GB csv file with ~330 columns (~7.5 million rows) into a SQL Server instance, but I'm doing the parsing internally as it the file has some quirks ...
0
votes
1
answer
75
views
Cannot bulk load because the file, BULK INSERT error code 3: The system cannot find the path specified
When I am using this code:
select @CHEMIN_FICHIER = 'C:\\crm_tmp\\Images\\Equipements\\' + @NOM_FICHIER;
it works well.
But this code is vulnerable to SQL injection, so I changed it to this:
select @...
0
votes
0
answers
21
views
When did EF Core include batch update natively and why is people still recommending 3rd options?
Back in the EF6 era, bulk updates or inserts were very slow because each changing entity triggered a round trip.
Last year when using EF Core 3.1, I found this feature where it automatically batches (...
-1
votes
1
answer
88
views
Bulk insert ,double Fieldterminator at the last column issue
when I insert from a csv with bulk, the last column has FIELDTERMINATOR twice when it should be null, in the csv that field is empty.
`
BULK INSERT my_table FROM 'path'
WITH (
...
0
votes
1
answer
90
views
What is the maximum query length accepted for a PostgreSQL multi-valued SELECT query?
The context is that I am writing a script to send load/insert query to a postgres server to insert rows into an existent table. I read many resources about the topic, about the approach to choose, ...
0
votes
0
answers
90
views
bulkCreate is not inserting JSON datatype field in mysql sequelize
I am trying to insert the bulk data via
bulkCreate
in nodejs application with sequelize. I want the following array of object (bulkData parameter value) to insert in the db
{ "name": "...
0
votes
1
answer
266
views
How to upload all files from a folder to S3 bucket of AWS?
This is the current code I am using to upload a single file from local folder to s3 bucket.
try {
var filePath = "./data/1061827.png";
var s3 = new AWS.S3();
var params =...
0
votes
1
answer
19
views
How to log _bulk api in Elastic Search
Currently we are using Elastic Search 5.4 and understand its deprecated version , but we have a requirement that we would need to log all the queries including the _bulk request as well , for some ...
0
votes
2
answers
62
views
Calling Stored Procedures is much slower than just calling insert and bulk insert is basically the same, Why?
I have a table and a stored procedure like following,
CREATE TABLE `inspect_call` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`task_id` bigint(20) unsigned NOT NULL DEFAULT '0',
`...
0
votes
0
answers
41
views
invalid column number in Formatfile sql
I looked at this one (Bulk inserting a csv in SQL using a formatfile to remove double quotes) but my situation is just different enough.
First, how would I upload this very lengthy format file? ...
0
votes
1
answer
34
views
Issues with bulk insert - file could not be found
I'm using this code
BULK INSERT equi2022a
FROM 'C:\Users\Someone\Documents\Processing\equi'
WITH (FORMATFILE = 'C:\Users\someone\Desktop\formatfileequi-2.txt'
I'm getting this error:
Msg 4860, ...
0
votes
0
answers
9
views
A service that makes API calls to extract data from a platform for each signed up user and then sends that data to my database
I want to create a service that makes a scheduled api call (every hour or so). This api call extracts the users data from a platform and then inserts that data to my database.
My question is what the ...
0
votes
0
answers
49
views
how to write procedure using bulk collect and for all method in oracle?
I have Billion's of records to updated and inserted to tables. i have created a procedure it's working but it's taking 10 min for 100k to updated, to reduce time i have used bulk collect and for all ...
0
votes
3
answers
199
views
Can I do a bulk insert into a table from Microsoft SQL Server Management Studio with a copy and paste list?
I often get a list of names I need to update in a table from an Excel list, and I end up creating a SSIS program to reads the file into a staging table and doing it that way. But is there I way I ...
0
votes
0
answers
134
views
how can I Ignore truncation error on bulk insert
Is there a way to ignore truncation?
If found this: https://www.sqlshack.com/sql-truncate-enhancement-silent-data-truncation-in-sql-server-2019/ but it doesn't work with bulk insert.
thanks!
0
votes
0
answers
79
views
input post array in CodeIgniter have a limit?
currently doing a bulk upload. After upload the excel file, user can validate the data then click submit to save. After user click submit, the row inserted in database only 43 rows. I have upload file ...
0
votes
0
answers
312
views
Nodejs Opensearch client bulk create not waiting all documents
I am new to Opensearch and I am trying to create a NodeJS class to add many asynchronous documents to an index; I read the bulk operations doc and thought that it covers my needs, especially because I ...
0
votes
0
answers
78
views
Can I fix the Bulk Insert of inconsistent CSV data using Format File?
I'm trying to load below CSV data to a SQL Server 2019 database table.
Though actual data is starting from Line: 7, yet I needed Line#3 (Valid from) and #4 (Valid to) as well, so I started from the ...