0

I'm trying to dump csv files into my table using cx_Oracle and python. But it's unbearably slowly (500 records in 336 secs). Kindly let me if there's any other method faster than this. Code is below

import pandas as pd
import cx_Oracle
import time


connection_string = '{}/{}@//{}:{}/{}'.format(user_name, password, host_name, port, service_name)
engine = cx_Oracle.connect(connection_string)


start_time = time.time()

t = pd.read_sql(con=engine, sql='select * from students where rownum < 18000')

print(t.shape)

t.to_sql(con=engine, name='students_new', if_exists='append', index=False)

print("Finished in : " + str(round(time.time() - start_time, 2)))
3
  • 1
    skip the python nonsense and use the csv as an oracle external table Commented Apr 1, 2020 at 21:32
  • @OldProgrammer i have to automate the process and python is my best bet Commented Apr 1, 2020 at 21:52
  • External tables are great, but need to be on the DB server machine, which isn't so practical for most people. Commented Apr 1, 2020 at 23:28

1 Answer 1

2

The example code you gave doesn't match the written question.

If you want to load data from as CSV file into Oracle database using Python, the straight cx_Oracle example is in the Loading CSV Files into Oracle Database section of the manual. You need to use executemany() to upload as much data as possible with each call to the database.

To cut & paste from the manual:

import cx_Oracle
import csv

. . .

# Predefine the memory areas to match the table definition
cursor.setinputsizes(None, 25)

# Adjust the batch size to meet your memory and performance requirements
batch_size = 10000

with open('testsp.csv', 'r') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    sql = "insert into test (id,name) values (:1, :2)"
    data = []
    for line in csv_reader:
        data.append((line[0], line[1]))
        if len(data) % batch_size == 0:
            cursor.executemany(sql, data)
            data = []
    if data:
        cursor.executemany(sql, data)
    con.commit()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.