I am working on a desktop application which uses SQLite to bulk insert tens of thousands of rows into a SQLite database. I would like help optimizing the bulk insert performance. It currently takes up to 50 seconds to insert 60 megs worth of data into the database.
what connection string paramaters could I use to improve performance? Should I change the buffer size? Is this possible via a connection string parameter? Are there any other connection string parameters to improve performance? My current connection string is:
Data Source=Batch.db;Version=3;Pooling=True;Max Pool Size=10;Synchronous=off;FailIfMissing=True;Journal Mode=Off;
I am using Dapper ORM. (built by the guys at StackOverflow) Is there a faster way to bulk insert into Sqlite, in .net?
System.Data.Sqlite is being used to insert into SQLite. What about getting a special compiled version of sqlite which improves performance? Is one version of SQLite better than another? Currently using System.Data.SQLite from http://sqlite.phxsoftware.com
Currently, I am wrapping inserts inside a transaction to make them faster (this made a good improvement).
I am inserting into one table at a time into 17 tables. Could I parallelize this on different threads and make this faster?
Current Performance. Is this typical? Can I do better?
- 55,000 rows into table with 19 columns: 2.25 sec to insert (24k inserts/sec)
- 10,000 rows into table with 63 columns: 2.74 sec to insert (3.7k/sec)
I like SQLite, but I would love to make it a bit faster. Currently saving my objects to an XML file using XML serialization is faster than saving to a SQLite database, so my boss is asking: why switch to SQLite? Or should I be using MongoDB, or some other object database?
PRAGMA
right and using a transaction. How this maps to the ORM though, I don't know.