62

I'm trying to bulk load a lot of data ( 5.5 million rows ) into an SQLite database file. Loading via INSERTs seems to be far too slow, so I'm trying to use the sqlite3 command line tool and the .import command.

It works perfectly if I enter the commands by hand, but I can't for the life of me work out how to automate it from a script ( .bat file or python script; I'm working on a Windows machine ).

The commands I issue at the command line are these:

> sqlite3 database.db
sqlite> CREATE TABLE log_entry ( <snip> );
sqlite> .separator "\t"
sqlite> .import logfile.log log_entry

But nothing I try will get this to work from a bat file or python script.

I've been trying things like:

sqlite3 "database.db" .separator "\t" .import logfile.log log_entry

echo '.separator "\t" .import logfile.log log_entry' | sqlite3 database.db

Surely I can do this somehow?

4
  • How slow is your loading via INSERTs? I insert 15 million large rows in Sqlite in 12 minutes with insert-statements. You have to use transactions and prepared statements.
    – tuinstoel
    Commented Mar 19, 2009 at 16:07
  • Very slow. I'm using python to parse a log file and insert each line as a row. I'm not using prepared statements, but I am using transactions. Even so it is much quicker to do this using the sqlite3 program.
    – dave
    Commented Mar 20, 2009 at 14:25
  • Using prepared statements makes a ginormous difference. I can create a two gigabyte sqlite database (1 big table with 1 index) in 12 minutes with INSERT's on a standard PC.
    – tuinstoel
    Commented Mar 27, 2009 at 11:30
  • @tuinstoel can you make example on syntax to place INSERTs prepared statements or transactions within an insert.sql script sqlite3 test.db < insert.sql? Here insert.sql contains CREATE TABLE log_entry ( <snip> ); .separator "\t" .import logfile.log log_entry as OP
    – user305883
    Commented Oct 12, 2016 at 13:40

8 Answers 8

63

Create a text file with the lines you want to enter into the sqlite command line program, like this:

CREATE TABLE log_entry (  );
.separator "\t"
.import logfile.log log_entry

and then just call sqlite3 database.db < commands.txt

1
  • 2
    You solution is correct as is, I got stuck as you cannot have any spaces before a dot . command in your script. FYI
    – ozmike
    Commented Oct 29, 2014 at 2:17
29

Alternatively you can put everything in one shell script file (thus simplifying maintenance) using heredoc import.sh :

#!/bin/bash --
sqlite3 -batch $1 <<"EOF"
CREATE TABLE log_entry ( <snip> );
.separator "\t"
.import logfile.log log_entry
EOF

...and run it:

import.sh database.db

It makes it easier to maintain just one script file. By the way, if you need to run it under Windows, Power Shell also features heredoc

In addition this approach helps to deal with lacking script parameter support. You can use bash variables:

#!/bin/bash --

table_name=log_entry

sqlite3 -batch $1 <<EOF
CREATE TABLE ${table_name} ( <snip> );
.separator "\t"
.import logfile.log ${table_name}
EOF

Or even do a trick like this:

#!/bin/bash --

table_name=$2

sqlite3 -batch $1 <<EOF
CREATE TABLE ${table_name} ( <snip> );
.separator "\t"
.import logfile.log ${table_name}
EOF

...and run it: import.sh database.db log_entry

1
  • The author - mentions he is working on a windows -FYI
    – ozmike
    Commented Oct 29, 2014 at 2:19
18

Create a separate text file containing all the commands you would normally type into the sqlite3 shell app:

CREATE TABLE log_entry ( <snip> );
.separator "\t"
.import /path/to/logfile.log log_entry

Save it as, say, impscript.sql.

Create a batch file which calls the sqlite3 shell with that script:

sqlite3.exe yourdatabase.db < /path/to/impscript.sql

Call the batch file.

On a side note - when importing, make sure to wrap the INSERTs in a transaction! That will give you an instant 10.000% speedup.

7

I just recently had a similar problem while converting Firefox' cookies.sqlite to a text file (for some downloading tool) and stumbled across this question.

I wanted to do that with a single shell line and that would be my solution applied to the above mentioned problem:

echo -e ".mode tabs\n.import logfile.log log_entry" | sqlite3 database.db

But I haven't tested that line yet. But it worked fine with the Firefox problem I mentioned above (btw via Bash on Mac OSX ):

echo -e ".mode tabs\nselect host, case when host glob '.*' then 'TRUE' else 'FALSE' end, path, case when isSecure then 'TRUE' else 'FALSE' end, expiry, name, value from moz_cookies;" | sqlite3 cookies.sqlite
3
  • I can confirm that this method works; just tried it on OSX with no problem.
    – Stan James
    Commented Jul 28, 2013 at 23:08
  • 1
    The author - mentions he is working on a windows -FYI
    – ozmike
    Commented Oct 29, 2014 at 2:20
  • There is no such command like ´echo -e´ under windows. Like ozmike already commented the author says "I'm working on a Windows machine".
    – PeterCo
    Commented Jan 16, 2017 at 14:52
6
sqlite3 abc.db ".read scriptname.sql"
2
  • 2
    Welcome to Stack Overflow! While this code snippet may solve the question, including an explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, not just the person asking now! Please edit your answer to add explanation, and give an indication of what limitations and assumptions apply. Commented Nov 15, 2016 at 17:13
  • Grateful for this answer despite the sparsity:) Elaboration on your answer here sqlitetutorial.net/sqlite-commands "To execute the SQL statements in the commands.txt file, you use the .read FILENAME command" Commented Oct 8, 2019 at 14:06
1

At this point, I'm not sure what else I can add other than, I had some trouble adding a unix environment variable to the bash script suggested by nad2000.

running this:

bash dbmake.sh database.db <(sed '1d' $DATA/logfile.log | head -n 1000)

I needed to import from stdin as workaround and I found this solution:

sqlite3 $1 <<"EOF"
CREATE TABLE log_entry;
EOF
sqlite3 -separator $'\t' $1 ".import $2 log_entry"

By adding the second sqlite3 line, I was able to pass the $2 from Unix into the file parameter for .import, full path and everything.

0

On Windows, this should work:

(echo CREATE TABLE log_entry ( <snip> ); & echo .separator "\t" & echo .import logfile.log log_entry) | sqlite3.exe database.db

I haven't tested this particular command but from my own pursuit of solving this issue of piping multiple commands I found that the key was to enclose the echoed commands within parentheses. That being said, it is possible that you may need to tweak the above command to also escape some of those characters. For example:

(echo CREATE TABLE log_entry ^( ^<snip^> ^); & echo .separator "\t" & echo .import logfile.log log_entry) | sqlite3.exe database.db

I'm not sure if the escaping is needed in this case, but it is highly probable since the parentheses may conflict with the enclosing ones, then the "less than" and "greater than" symbols are usually interpreted as input or output which may also conflict. An extensive list of characters' escape may be found here: http://www.robvanderwoude.com/escapechars.php

-1
   here trans is table name and trans.csv is a csv file in which i have 1959 rows of data

    $ sqlite3 abc.db ".separator ','"
    $ sqlite3 abc.db ".import 'trans.csv' trans"
    $ sqlite3 abc.db "select count(*) from trans;"
    1959

but its impossible to write like as you wrote

1
  • 1
    Calling ".separator ','" has no effect when you run multiple sqlite processes, as you have indicated above. Commented Oct 12, 2014 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.