167

I'm trying to import a csv file to an SQLite table.

Example csv:

1,2
5,6
2,7

Example command:

sqlite> create table foo(a, b);
sqlite> .separator ,
sqlite> .import test.csv foo
Error: test.csv line 1: expected 2 columns of data but found 4

I'm not even sure why it would find four columns with six pieces of data and two columns.

9
  • 1
    It appears that the command expects column headers in the first line, and that the line-terminator is not being recognized as such. 6 - 2 = 4 Feb 19, 2013 at 0:52
  • Can I specify a line-terminator or is there one missing from my csv? Feb 19, 2013 at 1:01
  • 1
    It doesn't look like you can specify the line-terminator from the SQLite command-line tool. Do you have a text-editor (like Notepad on Windows, but better) that will show you the line-terminating characters? There are three main variations: \r\n on Windows, \n on *nix (includes newer Macs), \r on older Macs. Feb 19, 2013 at 1:06
  • I'm using BBEdit, but I don't see any of those characters at the end of the lines. I tried adding them manually, but it doesn't seem to do anything... Feb 19, 2013 at 1:17
  • 1
    This website can convert a CSV (or Excel) file to SQLite - converttosqlite.com Dec 27, 2015 at 20:06

11 Answers 11

209

What also is being said in the comments, SQLite sees your input as 1, 25, 62, 7. I also had a problem with , and in my case it was solved by changing "separator ," into ".mode csv". So you could try:

sqlite> create table foo(a, b);
sqlite> .mode csv
sqlite> .import test.csv foo

The first command creates the column names for the table. However, if you want the column names inherited from the csv file, you might just ignore the first line.

--- New Versions of sqlite3 ---

The latest version of sqlite3 creates the table and columns for you if you let it.

You can also skip the line ".mode csv" if you append "--csv" to the end of the import statement like so:

sqlite> .import test.csv foo --csv

8
  • 112
    For other people who land here from a search, if the first line of your csv file contains the column names, then you can omit the first create table command and sqlite will use the column names from the csv file. Nov 3, 2015 at 20:56
  • 3
    @EarlCrapstone: could you elaborate? It doesn't seem to work for me.
    – d33tah
    Jan 14, 2016 at 7:58
  • 6
    @d33tah Look at these examples. Notice how the first line contains the column names Year,Make,Model,Description,Price and not actual data. If that's the case with your CSV file, then you do not need to manually create the table using the create table command. The .import command will use the first line in your file to determine the column names and create the table accordingly. You still have to include the name of the table in the command. The same info from the SQLite docs. Jan 14, 2016 at 13:16
  • 1
    Why are you setting .mode? Isn't that only for output?
    – Alan
    Jun 3, 2016 at 17:51
  • 1
    If you find that the generated table groups multiple columns together, check that you aren't using a reserved word such as type for a CSV column name.
    – paulvs
    Oct 15, 2016 at 18:05
46

The easiest is to add the comma-separated table headers directly to your CSV file, followed by a new line, and then all your CSV data.

In the SQLite shell enter:

$ sqlite3 yourfile.sqlite
sqlite>  .mode csv
sqlite>  .import test.csv yourtable
sqlite>  .exit
6
  • 2
    How to handle unescaped " character alerts?
    – TMOTTM
    Aug 8, 2017 at 19:52
  • 1
    If you create a table before importing, then it will not look for headers.
    – Rolf
    Feb 28, 2018 at 17:13
  • I have to put file in same folder where sqlite3.exe is located. Can i use another path for csv file ? Jun 11, 2019 at 4:03
  • Minor point: "If you haven't got Sqlite installed on your Mac" ... AFAIK it is installed by default. Feb 12, 2020 at 17:10
  • All columns are text columns. If quantitative data perhaps less work to type in the table definition beforehand, although the open source GUI tool "DB Browser for SQLite" can alter column types on the fly.
    – Lori
    Nov 21, 2021 at 17:56
34

Here's how I did it.

  • Make/Convert csv file to be seperated by tabs (\t) AND not enclosed by any quotes (sqlite interprets quotes literally - says old docs)
  • Enter the sqlite shell of the db to which the data needs to be added

    sqlite> .separator "\t" ---IMPORTANT! should be in double quotes sqlite> .import afile.csv tablename-to-import-to

2
  • for multiple rows, my tsv needed to specify the ROW separator with this command .separator "\t" "\r"
    – mfink
    May 13, 2016 at 21:03
  • 3
    When discussing file formats, tsv != csv See tools.ietf.org/html/rfc4180
    – Alan
    Jun 2, 2016 at 17:07
20

How to import csv file to sqlite3

  1. Create database

    sqlite3 NYC.db
    
  2. Set the mode & tablename

    .mode csv tripdata
    
  3. Import the csv file data to sqlite3

    .import yellow_tripdata_2017-01.csv tripdata
    
  4. Find tables

    .tables
    
  5. Find your table schema

    .schema tripdata
    
  6. Find table data

    select * from tripdata limit 10;
    
  7. Count the number of rows in the table

    select count (*) from tripdata;
    
9

before .import command, type ".mode csv"

Quote from Importing files as CSV or other formats

Note that it may be important to set the "mode" before running the ".import" command. This is prudent to prevent the command-line shell from trying to interpret the input file text as some format other than how the file is structured. If the --csv or --ascii options are used, they control import input delimiters.

And

To import data with arbitrary delimiters and no quoting, first set ascii mode (".mode ascii"), then set the field and record delimiters using the ".separators" command. This will suppress dequoting. Upon ".import", the data will be split into fields and records according to the delimiters so specified.

2
  • 5
    .mode is for output only Dec 15, 2015 at 12:41
  • 15
    Apparently not so. From the .import doc section: Note that it is important to set the "mode" to "csv" before running the ".import" command. This is necessary to prevent the command-line shell from trying to interpret the input file text as some other format. Sep 19, 2016 at 14:18
3

I had exactly same problem (on OS X Maverics 10.9.1 with SQLite3 3.7.13, but I don't think SQLite is related to the cause). I tried to import csv data saved from MS Excel 2011, which btw. uses ';' as columns separator. I found out that csv file from Excel still uses newline character from Mac OS 9 times, changing it to unix newline solved the problem. AFAIR BBEdit has a command for this, as well as Sublime Text 2.

2
  • 3
    It's easier to use tr -s '\r' '\n' and that will work with all sorts of random files. Jan 12, 2014 at 18:05
  • sed, tr, and other unix text processing tools are perfect for this job. They already exist on OSX, Linux, BSD, Plan9, and WSL. Not everyone is going to have Sublime or OSX and should be able to use the tools provided by the OS. Oct 10, 2023 at 3:32
3

With you can do it in one line:

termsql -i mycsvfile.CSV -d ',' -c 'a,b' -t 'foo' -o mynewdatabase.db

From Termsql

Convert text from a file or from stdin into SQL table and query it instantly. Uses sqlite as backend. The idea is to make SQL into a tool on the command line or in scripts.

Termsql requires and

2
  • 1
    link was broken, changed it to the github repo - please clarify!
    – user57508
    Nov 17, 2015 at 15:47
  • please explain why readers should use yet another two tools as the question is originally about sqlite itself and that sqlite can parse standard SQL directly from STDIN (e.g a unix pipe such as echo .import mydata.csv foo --csv | sqlite3 mynewdatabase.db). Oct 10, 2023 at 3:30
2

In my case I had to see the file I was trying to add, it had headers and the separator was a semicolon ;.

I tried first:

.mode csv
.import myfile.csv mytable

But it didn't work, so I tried:

.separator ";"
.import myfile.csv mytable

And it did work, so I had to set the separator manually.

1

As some websites and other article specifies, its simple have a look to this one. https://www.sqlitetutorial.net/sqlite-import-csv/

We don't need to specify the separator for csv file, because csv means comma separated.

sqlite> .separator , no need of this line.

sqlite> create table cities(name, population);
sqlite> .mode csv
sqlite> .import c:/sqlite/city_no_header.csv cities

This will work flawlessly :)

PS: My cities.csv with header.


name,population
Abilene,115930
Akron,217074
Albany,93994
Albuquerque,448607
Alexandria,128283
Allentown,106632
Amarillo,173627
Anaheim,328014
1
  • 2
    Don't forget to run .save someFileName in the end. Took me hour to find this to finally export this in memory database file to disk 😅
    – ioopl
    May 7, 2020 at 0:24
0

Follow the steps:-

1] sqlite3 name

2] .mode csv tablename

3] .import Filename.csv tablename

0

In my case i had a similar problem the lines in the csv file did contain a \n instead of an actual a CRLF (line break).

     Alias, Name, Agency\n
     007  , James Bond, MI5\n
     Q    , Mister Q, MI6\n
     Moneypenny, Miss Moneypenny, MI5\n

From Common Format and MIME Type for Comma-Separated Values (CSV) Files

  1. Each record is located on a separate line, delimited by a line break (CRLF). For example:

    aaa,bbb,ccc CRLF

    zzz,yyy,xxx CRLF

  2. The last record in the file may or may not have an ending line break. For example:

    aaa,bbb,ccc CRLF

    zzz,yyy,xxx

After i replaced the line separators \n with a line break (using SHIFT+ENTER in vs code) the import under windows was possible using

c:\Apps\SQLite\sqlite_3.40.1>sqlite3.exe
                SQLite version 3.40.1 2022-12-28 14:03:47
                Enter ".help" for usage hints.
                Connected to a transient in-memory database.
                Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open C:/csv/myDataCRLF.db
sqlite> .import --csv C:/csv/source_file_crlf.csv myTable
sqlite> .save C:/csv/myDataCRLF.db
                Error: database is locked
sqlite> .save C:/csv/myDataCRLF_2.db

The database file myDataCRLF_2.db has now a table myTable with the records from the csv-file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.