138

For development I'm using SQLite database with production in PostgreSQL. I updated my local database with data and need to transfer a specific table to the production database.

Running sqlite database .dump > /the/path/to/sqlite-dumpfile.sql, SQLite outputs a table dump in the following format:

BEGIN TRANSACTION;
CREATE TABLE "courses_school" ("id" integer PRIMARY KEY, "department_count" integer NOT NULL DEFAULT 0, "the_id" integer UNIQUE, "school_name" varchar(150), "slug" varchar(50));
INSERT INTO "courses_school" VALUES(1,168,213,'TEST Name A',NULL);
INSERT INTO "courses_school" VALUES(2,0,656,'TEST Name B',NULL);
....
COMMIT;

How do I convert this into a PostgreSQL compatible dump file I can import into my production server?

1
  • 5
    Well, that command did not work for me until I changed sqlite to sqlite3 Mar 25, 2018 at 19:57

9 Answers 9

136

You should be able to feed that dump file straight into psql:

/path/to/psql -d database -U username -W < /the/path/to/sqlite-dumpfile.sql

If you want the id column to "auto increment" then change its type from "int" to "serial" in the table creation line. PostgreSQL will then attach a sequence to that column so that INSERTs with NULL ids will be automatically assigned the next available value. PostgreSQL will also not recognize AUTOINCREMENT commands, so these need to be removed.

You'll also want to check for datetime columns in the SQLite schema and change them to timestamp for PostgreSQL. (Thanks to Clay for pointing this out.)

If you have booleans in your SQLite then you could convert 1 and 0 to 1::boolean and 0::boolean (respectively) or you could change the boolean column to an integer in the schema section of the dump and then fix them up by hand inside PostgreSQL after the import.

If you have BLOBs in your SQLite then you'll want to adjust the schema to use bytea. You'll probably need to mix in some decode calls as well. Writing a quick'n'dirty copier in your favorite language might be easier than mangling the SQL if you a lot of BLOBs to deal with though.

As usual, if you have foreign keys then you'll probably want to look into set constraints all deferred to avoid insert ordering problems, placing the command inside the BEGIN/COMMIT pair.

Thanks to Nicolas Riley for the boolean, blob, and constraints notes.

If you have ` on your code, as generated by some SQLite3 clients, you need to remove them.

PostGRESQL also doesn't recognize unsigned columns, so you might want to drop that or add a custom-made constraint such as this:

CREATE TABLE tablename (
    ...
    unsigned_column_name integer CHECK (unsigned_column_name > 0)
);

While SQLite defaults null values to '', PostgreSQL requires them to be set as NULL.

The syntax in the SQLite dump file appears to be mostly compatible with PostgreSQL so you can patch a few things and feed it to psql. Importing a big pile of data through SQL INSERTs might take a while but it'll work.

20
  • 4
    No, you want to keep the transaction to avoid some overhead. Jan 3, 2011 at 7:07
  • 3
    This works great. I would also note that if you need to migrate sqlite datetime columns, that you have to change them to timestamp for postgres.
    – Clay
    Feb 6, 2012 at 15:56
  • 4
    A few more issues I ran into: changing BLOB into BYTEA (stackoverflow.com/questions/3103242), changing 0/1 for BOOLEAN columns to '0'/'1', and deferring constraints (DEFERRABLE / SET CONSTRAINTS ALL DEFERRED). Feb 11, 2012 at 0:47
  • 2
    @NicholasRiley: Thanks for that. I gave this up to a community wiki since it has turned into a group effort, fair is fair. Feb 11, 2012 at 5:10
  • 2
    You can use to_timestamp() in the postgreSQL to convert a timestamp to a progreSQL timestamp
    – roeland
    Mar 23, 2013 at 10:51
90

pgloader

I came across this post when searching for a way to convert an SQLite dump to PostgreSQL. Even though this post has an accepted answer (and a good one at that +1), I think adding this is important.

I started looking into the solutions here and realized that I was looking for a more automated method. I looked up the wiki docs:

https://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL

and discovered pgloader. Pretty cool application and it's relatively easy to use. You can convert the flat SQLite file into a usable PostgreSQL database. I installed from the *.deb and created a command file like this in a test directory:

load database  
    from 'db.sqlite3'  
    into postgresql:///testdb 
       
with include drop, create tables, create indexes, reset sequences  
         
set work_mem to '16MB', maintenance_work_mem to '512 MB';

like the docs state. I then created a testdb with createdb:

createdb testdb

I ran the pgloader command like this:

pgloader command

and then connected to the new database:

psql testdb

After some queries to check the data, it appears it worked quite well. I know if I had tried to run one of these scripts or do the stepwise conversion mentioned herein, I would have spent much more time.

To prove the concept I dumped this testdb and imported into a development environment on a production server and the data transferred over nicely.

4
  • 3
    Beware that (still supported) Ubuntu distributions might have outdated version - v2.x.y are already deprecated and don't actually work. v3.2.x might work but v3.2.3 is recommended. I have fetched v3.2.3 from bleeding edge and installed with sudo dpkg -i <.deb file name>, it had no problem with dependencies.
    – silpol
    May 21, 2016 at 17:03
  • 1
    I concur with @silpol - be sure to download the latest stable release and install using your fav package manager; for the "command" file this is just a text file called 'command' with no extension name (i.e. no need for .txt at the end of the file name) you don't need to put the file name in angular brackets; i had to change the search_parth of the psql database in order to see my data; pgloader works well and saved me a great deal of hassle
    – BenKoshy
    Sep 29, 2016 at 20:51
  • 1
    Yeah, I was struggling when I encountered this issue, and that tool made it so easy... Sometimes things just work out nicely, don't they? Apr 29, 2017 at 15:04
  • Unfortunately it does not work on windows.
    – sveri
    Feb 17, 2021 at 9:12
32

The sequel gem (a Ruby library) offers data copying across different databases: http://sequel.jeremyevans.net/rdoc/files/doc/bin_sequel_rdoc.html#label-Copy+Databases

First install Ruby, then install the gem by running gem install sequel.

In case of sqlite, it would be like this: sequel -C sqlite://db/production.sqlite3 postgres://user@localhost/db

6
  • 4
    Awesome solution. Much easier than fiddling around with pgloader. Oct 26, 2019 at 12:10
  • 2
    Absolutely, pgloader is messy, the GC seems to crash on huge databases: github.com/dimitri/pgloader/issues/962
    – hasufell
    Oct 31, 2019 at 8:39
  • Feel free to post your answer at stackoverflow.com/questions/6148421/… where I copied your answer. Then ping me and I will revoke my answer if you want the reps for it.
    – Felix
    Oct 6, 2020 at 8:38
  • @Felix thanks! You can take the credit. Could you swap the order of DB references around (since it wants PG to SQLite), oh and add one more "la" to my id. The answer may also be less helpful though since it requires them installing PG on dev machine, and at that point they'd just use PG for development.
    – lulalala
    Oct 7, 2020 at 1:39
  • @lulalala Thanks. Did that. But about the reasoning I disagree. They could e.g. convert the db on the linux machine and then copy it over to the dev machine (as sqlite db file). But anyway all in all its a bad idea :) But sequel saved my ass here in a nasty situation.
    – Felix
    Oct 7, 2020 at 7:25
18

You can use a one liner, here is an example with the help of sed command:

sqlite3 mjsqlite.db .dump | sed -e 's/INTEGER PRIMARY KEY AUTOINCREMENT/SERIAL PRIMARY KEY/g;s/PRAGMA foreign_keys=OFF;//;s/unsigned big int/BIGINT/g;s/UNSIGNED BIG INT/BIGINT/g;s/BIG INT/BIGINT/g;s/UNSIGNED INT(10)/BIGINT/g;s/BOOLEAN/SMALLINT/g;s/boolean/SMALLINT/g;s/UNSIGNED BIG INT/INTEGER/g;s/INT(3)/INT2/g;s/DATETIME/TIMESTAMP/g' | psql mypqdb mypguser
8
  • there is no replace for LONG type, e.g. Jul 11, 2015 at 23:12
  • 2
    one more item could be added sed -e 's/DATETIME/TIMESTAMP/g'
    – silpol
    May 17, 2016 at 23:53
  • 1
    sed -e 's/TINYINT(1)/SMALLINT/g' -- and for a comparison of all the data types see stackoverflow.com/questions/1942586/… Jun 28, 2017 at 22:07
  • I also had a problem with a SMALLINT that defaulted to 't' or 'f' in the sqlite. Obviously a boolean, but not familiar enough with either db system to recommend a safe fix.
    – labyrinth
    Apr 3, 2018 at 20:44
  • 2
    Replace ' | sed -e ' with ; :)
    – AstraSerg
    Dec 3, 2019 at 8:32
17

I wrote a script to do the sqlite3 to postgres migration. It doesn't handle all the schema/data translations mentioned in https://stackoverflow.com/a/4581921/1303625, but it does what I needed it to do. Hopefully it will be a good starting point for others.

https://gist.github.com/2253099

1
5

Try these steps...

Step 01: Dump sqlite db to json

python3 manage.py dumpdata > data.json

Step 02: Create tables without migration

python3 manage.py migrate --run-syncdb

Step 03: Open django shell. Then exclude contentype data

python3 manage.py shell
from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
quit()

Step 04: Load Data

python3 manage.py loaddata data.json
2
  • 1
    for large GB JSON files huge ram required
    – giveJob
    Dec 6, 2020 at 5:51
  • Thanks a million for this. Why should content types be excluded? Jan 25 at 14:03
2

pgloader work wonders on converting database in sqlite to postgresql.

Here's an example on converting a local sqlitedb to a remote PostgreSQL db:

pgloader sqlite.db postgresql://username:password@hostname/dbname

1
  • 3
    Pgloader is terribly buggy and unreliable. It immediately crashes with the error KABOOM! Control stack exhausted (no more space for function call frames).
    – Cerin
    Feb 1, 2019 at 22:43
1

I have tried editing/regexping the sqlite dump so PostgreSQL accepts it, it is tedious and prone to error.

What I got to work really fast:

First recreate the schema on PostgreSQL without any data, either editing the dump or if you were using an ORM you may be lucky and it talks to both back-ends (sqlalchemy, peewee, ...).

Then migrate the data using pandas. Suppose you have a table with a bool field (which is 0/1 in sqlite, but must be t/f in PostgreSQL)

def int_to_strbool(df, column):
    df = df.replace({column: 0}, 'f')
    df = df.replace({column: 1}, 't')
    return df

#def other_transform(df, column):
#...

conn = sqlite3.connect(db)
df = pd.read_sql(f'select * from {table_name}', conn)

df = int_to_strbool(df, bool_column_name)
#df = other_transform(df, other_column_name)

df.to_csv(table_name + '.csv'), sep=',', header=False, index=False)

This works like a charm, is easy to write, read and debug each function, unlike (for me) the regular expressions.

Now you can try to load the resulting csv with PostgreSQL (even graphically with the admin tool), with the only caveat that you must load the tables with foreign keys after you have loaded the tables with the corresponding source keys. I did not have the case of a circular dependency, I guess you can suspend temporarily the key checking if that is the case.

0

SyncLite (https://www.synclite.io) provides SQLite connector for numerous instances of applications each creating several SQLite databases, to perform a real-time database replication/consolidation into one or more databases/data warehouses/data lakes of your choice, while maintaining live replicas of source SQLite dbs. This gives rise to many interesting architectural possibilities for building all kinds of data intensive applications around SQLite.

Additionally,the SyncLite database replication/migration tool provides an ability to replicate/migrate data from pre-existing SQLite database files into a wide range of DBs/DWs/DataLakes.

1
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review
    – GuedesBF
    Dec 23, 2023 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.