I want to upload a huge number of entries (~600k) into a simple table in a PostgreSQL DB, with one foreign key, a timestamp and 3 float per each entry. However, it takes 60 ms per each entry to execute the core bulk insert described here, thus the whole execution would take 10 h. I have found out, that it is a performance issue of executemany()
method, however it has been solved with the execute_values()
method in psycopg2 2.7.
The code I run is the following:
#build a huge list of dicts, one dict for each entry
engine.execute(SimpleTable.__table__.insert(),
values) # around 600k dicts in a list
I see that it is a common problem, however I have not managed to find a solution in sqlalchemy itself. Is there any way to tell sqlalchemy to call execute_values()
in some occasions? Is there any other way to implement huge inserts without constructing the SQL statements by myself?
Thanks for the help!
SimpleTable.__table__.insert().values(values)
, which would compile to a single INSERT statement with multiple VALUES tuples, but it turned out that it was actually even slower on my machine. The compiling itself was as slow as using your method that relies onexecutemany()
.