97

This is a follow up question to a previously answered post: Is there a command line utility for validating SQLite databases in Linux?

If a database is producing the following error:

$ sqlite3 mydata.db "PRAGMA integrity_check"
Error: database disk image is malformed

Is there any cleanup processing that can be done to recover the database to a usable state? Even at the potential loss of some corrupt records?

16 Answers 16

160

UPDATE: There is now an automatic method that is built into SQLite: .recover

Sometimes, the corruption is only or mostly in indexes, in which case it would be possible to get some or most records by trying to dump the entire database with .dump, and use those commands to create a new database:

$ sqlite3 mydata.db ".dump" | sqlite3 new.db

However, this is not always possible.

The easiest and most reliable way is to restore the database file from the backup.

9
  • 3
    This actually worked for me. ".dump" produced acceptable SQL from my corrupt file that integrity_check was failing on. Nov 26, 2014 at 17:15
  • 3
    Huge thanks for your reply! Fixed my database in minutes ;) It is, however, important to note, that all entries that contain malformed data will simply not be copied, and the new database will contain all entries that could be recovered from the old database. Just take a look at the output of .dump, you'll see how your database works and which entries produce errors ;)
    – LukeLR
    Sep 10, 2015 at 15:21
  • I just gave this a try with a similar issue and it gave me an empty file and no feedback. Without the "| sqlite3 new.db" part it gives me feedback, but no file. Help anyone?
    – Ne Mo
    Jun 4, 2017 at 12:16
  • 1
    @NeMo I also got an empty db but then when I looked at the script the last line was ROLLBACK; I removed this and changed it to COMMIT; which worked
    – RobbZ
    Aug 15, 2019 at 21:49
  • 4
    @NeMo The command as written by CL is 2 parts. The first part creates a SQL script of the entire DB (.dump). The second half writes it to a new db. But because it is piping the output of one command to the other you never get an actual file. If you pipe the output of the first command to a file you will be able to edit it. So if you instead run sqlite3 mydata.db ".dump" > db_script.sql you will get a SQL script you can edit and make sure there is nothing strange at the end like ROLLBACK; Then you can create a database from that script using sqlite3 recovered.db < db_script.sql
    – RobbZ
    Aug 16, 2019 at 11:41
63

With Sqlite 3.29.0 a new .recover command has been introduced to the CLI:

Add the ".recover" command which tries to recover as much content as possible from a corrupt database file.

sqlite3 broken.db ".recover" | sqlite3 new.db
5
  • The command does not work, it shows "SQL error: database disk image is malformed" .. i know that its corrupted, im trying to recover Sep 23, 2019 at 8:45
  • @SubqueryCrunch I shared what I came across. I hope SQLite guys will see your message. I have come across this "malformed" error when my first page itself was corrupted. Sep 23, 2019 at 12:02
  • 4
    This worked for me (though I ran out of memory). I used sqlite3 broken.db ".recover" | sqlite3 new.db
    – Luca
    Dec 12, 2019 at 13:42
  • 2
    ".recover" worked on a >1Gb database for me. It only took 1min to recover the database so It is more efficient than any commercial SQLite recovery tools I tried before (some of which where not even able to recover the database while others took nearly an hour to do it). So big thanks to the SQLite developper team for this really usefull feature Jan 15, 2020 at 12:36
  • I tried this. Error messages: "Error: near line 128760: too many columns on lost_and_found" followed by many lines of "Error: near line 123456: no such table: lost_and_found". Not sure what's going on here... Jun 15, 2021 at 10:42
25

I had an sqlite file that was corrupt that would show a symptom like this.

select count(*) from corruptTable;
return:38000;

But when I would try to load the records with

select * from corruptTable;

It would only return 7 records.

I tried several things, but these steps were the most successful.

On a mac, open terminal and run these commands on your corrupt database. (these are sqlite3 commands, so you should be able to use other sqlite3 editors or similar commands in other systems).

1 sqlite3 dbWithCorruptTable.sqlite (Obviously replace "dbWithCorruptTable" to your sqlite3 file that has the corrupt table)
2 .mode insert
3 .output dump_all.sql
4 .dump
5 .exit
6 Manually edit the dump_all.sql file in a text editor and remove the transaction statements. Usually there is a "BEGIN TRANSACTION" statement on the 2nd line of the file and a "ROLLBACK" statement on the last line. Remove these and save the file

These steps were taken from this website: http://www.dosomethinghere.com/2013/02/20/fixing-the-sqlite-error-the-database-disk-image-is-malformed/

5
  • 5
    Note: when the file is very big (like 500Mb) you can't easily edit the SQL file. you should then use a command line this : --> cat dump_all_last.sql|grep -v TRANSACTION|grep -v ROLLBACK >dump_all_last_notrans.sql
    – Nadir
    Sep 11, 2015 at 2:18
  • 1
    Good Point Nadir, thanks for adding to this with your comment.
    – johnrechd
    Sep 11, 2015 at 15:19
  • Do you have any idea how to get this into an Automator app?
    – callisto
    Apr 28, 2016 at 10:38
  • 1
    This approach may be good for a small database, but on a 8.5GB database it produces about 15 GB of SQL code. Even the Visual Studio editor can't open this file for editing on my machine.
    – Mar
    Oct 11, 2016 at 6:54
  • Just a suggestion for those that have a DB that is too big to read into an editor. Just write a simple file parsing program that reads the file and looks for the transaction statements and removes them (or writes each line to a new file and excludes those). Shouldn't be the worst thing to implement.
    – johnrechd
    Sep 10, 2019 at 16:48
16

My method is similar, prevents a error rollback script:

sqlite3 database.db ".dump" | sed -e 's|^ROLLBACK;\( -- due to errors\)*$|COMMIT;|g' | sqlite3 database.new
2
  • 2
    This was the only method that allowed me to repair the scheme of my nextcloud sqlite db, so I could restore a backup (which requires the main script to run, which early schema errors will prevent).
    – TauPan
    Oct 21, 2019 at 17:03
  • 1
    This worked me to restore the history of the AWS CLI which, due to the corrupt history db constantly errors out with database disk image is malformed. Nov 21, 2019 at 16:15
12

If the database is seriously corrupt, the .dump will contain errors, and some data may be lost.

For more complex data schemas, this will mean orphaned and/or partial records which may confuse the application.

It may be preferable to .dump to a file, then use a text editor to remove problematic rows. Search for ERROR within the dump file.

1
  • 1
    Passing the dump directly to sqlite3 was not working alone, removing error lines in temporary dump fixed it. Aug 31, 2015 at 18:51
9

I was able to repair my Chrome history file (which is a sqlite3 database file) this way:

sqlite3.exe History ".backup History-new"
1
  • 3
    This is nice! I've always used .dump and then loaded to a new database. Your method has the upside that it works with databases that use custom collations etc things that don't with .dump.
    – mz2
    Feb 20, 2017 at 16:53
7

I know this is an old question, but I would still like to share my solution. My problem was that a sqlite3 database for kodi(xbmc) was corrupted.

.dump did not work in my case

file is encrypted or is not a database

What worked was the following:

  1. Made a backup of the old db File
  2. Let kodi create a new db File
  3. Checked on this site for the header format of sqlite files
  4. Opened both files with a hex editor (bless) and checked the first 96 Bytes
  5. The first 40 bytes where different so i copied the first 40 bytes from the new db file to the old db file
  6. After doing this, my database file worked again !!!
1
  • 2
    Nice one. After accomplishing this, you should probably immediately rebuild the database, .e.g. using .dump/.repair as in other answers, to make sure the file has complete integrity.
    – mwfearnley
    Feb 21, 2020 at 16:31
6

The pragma writable_schema disables some integrity checks, so this two commands might also do the trick, keeping db customizations in place:

PRAGMA writable_schema=ON;
VACUUM;
1
  • 1
    This allowed me to recover a very corrupted database! Cheers! Sep 23, 2019 at 9:27
4

I have fixed database corruption caused by missing indexes with these steps and they are working for me.

  1. DROP Index: sqlite drop index command

  2. Run vacuum Sqlite vacuum command

  3. Recreate index again : Sqlite create index

3

This worked for me:

Download the sqlite3 tools package from here and put it into any folder. Put your corrupted database in the same folder.

Open a command prompt.

Type the following:

sqlite3.exe

(Press Enter)

NAME_OF_YOUR_CORRUPTED_DATABASE> ".dump" | sqlite3 new.db

(Press enter)

All the other solutions didn't work for me.

1
  • 1
    This looks like an interactive version of the accepted answer? (Albeit obviously Windows-specific..)
    – mwfearnley
    Feb 21, 2020 at 16:34
3

If the error from PRAGMA integrity_check is like "row NNN missing from index sqlite_autoindex_XXX", you can fix indexes with command REINDEX.

2

The following fix worked to repair my database without running any command line tools.

I got the "database disk image is malformed" error message when I was working with one of my tables so I ran [PRAGMA integrity_check] which returned

Main freelist: free-page count in header is too small

On tree page 16198 cell 1: 2nd reference to page 14190

Page 16988 is never used

Page 46637 is never used

row 4493 missing from index indexname1

row 4493 missing from index indexname2

row 4493 missing from index indexname3

I first saved the schema for the table with the bad indexes so I could recreate those indexes. I then dropped the indexname 1, 2, and 3 indexes with the [drop index _] command. I exported my tables one by one to JSON files and then truncated each table. Running the integrity check at that point was successful. I then added the three indexes back with the [create index _] command and imported each table's records from their JSON file export. Running the integrity check command is still returning "ok" with all of the records restored.

2

I fixed it with the following steps after I even could not remove single corrupt rows by sql statement, only all.

  1. Open the *.db with sqlite browser and PRAGMA integrity_check; indicates the corrupt table.
  2. export this table as csv by sqlite browser
  3. delete all in table. DELETE FROM [tablename]
  4. Import csv with sqlite browser and select in import settings -> Advanced -> Conflict strategy -> Ignore Rows
  5. After that my Integrity Check was OK
2

Open sqlite database in any db browser tool. Then issue following commands

vacuum;
REINDEX;

And save the changes. Mostly this will solve the error.

0

i had a similar problem:
i wanted to parse an incomplete (partially downloaded) sqlite3 database file.
i came up with a custom sqlite3 parser, written in python, based on kaitai_struct

example use:

prepare.sh

sqlite3 example.db '
  create table t1 (a INT);
  create table t2 (a INT);
  insert into t1 values (1);
  insert into t2 values (2);
'

du -b example.db
# 12288   example.db

sqlite3 example.db 'pragma page_size'
# 4096

expr 12288 / 4096
# 3

# remove page 3
dd if=example.db of=example-partial.db bs=4096 count=2

sqlite3 example-partial.db 'select * from t1'
# Error: in prepare, database disk image is malformed (11)

parse.py

import pysqlite3

database = "example-partial.db"

con = pysqlite3.connect(database, allow_bad_size=True)

print(f"page size: {con._db.header.page_size} bytes")
print(f"db size: {con._db.header.num_pages} pages")
print(f"idx_lock_byte_page: {con._db.header.idx_lock_byte_page}")
print(f"idx_first_ptrmap_page: {con._db.header.idx_first_ptrmap_page}")
print(f"idx_last_ptrmap_page: {con._db.header.idx_last_ptrmap_page}")
print(f"con._db.pages =", con._db.pages)
print("tables =", con._tables)

print("con._columns")
for table in con._tables:
    try:
        print(f"table {table}: columns =", con._columns(table))
    except NotImplementedError as err:
        print("ignoring NotImplementedError:", err)

def format_values(values):
    result = []
    max_len = 50
    for val in values:
        s = repr(val)
        if len(s) > max_len:
            s = s[0:max_len] + "..."
        result.append(s)
    return "[" + ", ".join(result) + "]"

print("con._table_values")
for table in con._tables:
    print(f"con._table_values(table={repr(table)})")
    num_rows = 0
    for row_id, values in enumerate(con._table_values(table)):
        num_rows += 1
        print(f"table {table}: row {row_id + 1} =", format_values(values))

output of parse.py

warning: bad size. expected 12288. actual 8192
page size: 4096 bytes
db size: 3 pages
idx_lock_byte_page: 262144
idx_first_ptrmap_page: 0
idx_last_ptrmap_page: -1
con._db.pages = <pysqlite3.parser.sqlite3_helpers.PagesList object at 0x7fe44a4ee230>
tables = ['t1', 't2']
con._columns
table t1: columns = ['a']
table t2: columns = ['a']
con._table_values
con._table_values(table='t1')
table t1: row 1 = [1]
con._table_values(table='t2')
EOFError: requested 1 bytes, but only 0 bytes available

the EOFError is expected, because page 3 is missing, which contains the table t2

0

DB Browser Export data only

import data to new database

save

close

enter image description here

Not the answer you're looking for? Browse other questions tagged or ask your own question.