28

If I have some sample data, how do I put it into SQLite (preferably fully automated)?

{"uri":"/","user_agent":"example1"}
{"uri":"/foobar","user_agent":"example1"}
{"uri":"/","user_agent":"example2"}
{"uri":"/foobar","user_agent":"example3"}

Clarification to this post as requested via comments and as per the original requirements:

  • the data should be contained in an SQLite database such that the database file contains the entire data and can be copied or archived on its own while retaining any content (for instance to allow one to verify results)
  • the data contained should be structured, meaning queries on individual fields of the original JSON should be possible (think SELECT * WHERE uri = '/'), ideally with index support to make it work with large files
  • in an ideal case "fully automated" describes that a single script, one-liner, program, or similar can be executed to convert the file; this explicitly allows multi-step conversions as long as they are scriptable (several chained commands) but rejects any solutions that require manually filling in data such as the key/column names (JSON/SQL terminology respectively)
  • the file in this specific instance is an aggregation of log entries from a multitude of webservers, meaning the resulting file was about 80G in size (so any conversions running in-memory are problematic), although it was generated only once. Solutions which also allow to insert/import/append new data into the database and do not require the entire database to be recreated would be preferred
  • the file may be assumed to be local, however solutions which allow the file to be streamed from a different location (for instance by means of ssh server cat file | fancy_conversion_command) would be preferred
2
  • 5
    For search sake, that is JSONL (JSON lines) format, not JSON.
    – Leo
    Sep 12, 2019 at 3:56
  • What do you mean by "put it into SQLite". Put it in how? What is your table structure? What counts as fully automated? Where is the file coming from and how often does it appear? Is it being placed on the server where the SQLite database exists, or a different one? A significant number of additional details are needed to answer this question without making assumptions that could be wrong.
    – TylerH
    Mar 6 at 16:16

9 Answers 9

38

A way do this without CSV or a 3rd party tool is to use the JSON1 extension of SQLite combined with the readfile extension that is provided in the sqlite3 CLI tool. As well as overall being a "more direct" solution, this has the advantage of handling JSON NULL values more consistently than CSV, which will otherwise import them as empty strings.

If the input file is a well-formed JSON file, e.g. the example given as an array:

[
{"uri":"/","user_agent":"example1"},
{"uri":"/foobar","user_agent":"example1"},
{"uri":"/","user_agent":"example2"},
{"uri":"/foobar","user_agent":"example3"}
]

Then this can be read into the corresponding my_table table as follows. Open the SQLite database file my_db.db using the sqlite3 CLI:

sqlite3 my_db.db

then create my_table using:

CREATE TABLE my_table(uri TEXT, user_agent TEXT);

Finally, the JSON data in my_data.json can be inserted into the table with the CLI command:

INSERT INTO my_table SELECT 
  json_extract(value, '$.uri'), 
  json_extract(value, '$.user_agent')
FROM json_each(readfile('my_data.json'));

If the initial JSON file is newline separated JSON elements, then this can be converted first using jq using:

jq -s <my_data_raw.json >my_data.json

It's likely there is a way to do this directly in SQLite using JSON1, but I didn't pursue that given that I was already using jq to massage the data prior to import to SQLite.

2
  • 1
    Please note that jq -s <my_data_raw.json >my_data.json will consume quite some memory depending on how big a database you want to convert. I've been working with databases well above 30G, in which case this would require much more than the device at hand could provide. I'll add a word or two on that in my answer.
    – benaryorg
    Aug 2, 2020 at 10:55
  • 1
    @benaryorg good point. And I imagine there's a relatively easy way to do this directly in SQLite as their JSON1 extension is SO powerful despite it's seeming simplicity. I'm swamped with other things at the moment but I welcome an addition to avoid the use of jq completely.
    – mm2001
    Aug 3, 2020 at 11:32
37

I found the easiest way to do this is by using jq and CSV as an intermediary format.

Getting the CSV

First write your data to a file. I will assume data.json here.

Then construct the header using jq:

% head -1 data.json | jq -r 'keys_unsorted | @csv'
"uri","user_agent"

The head -1 is because we only want one line. jq's -r makes the output a plain string instead of a JSON-String wrapping the CSV. We then call the internal function keys_unsorted to get the keys of the input as an array, without sorting applied. This we send to the @csv formatter which outputs us a single string with the headers in quoted CSV format.

We then need to construct the data.

% jq -r 'map(tostring) | @csv' < data.json
"/","example1"
"/foobar","example1"
"/","example2"
"/foobar","example3"

We now take the whole input and deconstruct the associative array (map) using .[] and then put it back into a simple array […]. This basically converts our dictionary to an array of keys. Sent to the @csv formatter, we again get some CSV.

Putting it all together we get a single one-liner in the form of:

% (head -1 data.json | jq -r 'keys_unsorted | @csv' && jq -r 'map(tostring) | @csv' < data.json) > data.csv

If you need to convert the data on the fly, i.e. without a file, try this:

% cat data.json | (read -r first && jq -r '(keys_unsorted | @csv),(map(tostring) | @csv)' <<<"${first}" && jq -r 'map(tostring) | @csv')

Loading it into SQLite

Open an SQLite database:

sqlite3 somedb.sqlite

Now in the interactive shell do the following (assuming you wrote the CSV to data.csv and want it in a table called my_table):

.mode csv
.import data.csv my_table

Now close the shell and open it again for a clean environment. You can now easily SELECT from the database and do whatever you want to.

Putting it all together

Have an asciinema recording right there:

asciicast

Edits

Edit: As pointed out (thanks @Leo), the original question did show newline delimited JSON objects, which each on their own conform to rfc4627, but not all together in that format. jq can handle a single JSON array of objects much the same way though by preprocessing the file using jq '.[]' <input.json >preprocessed.json. If you happen to be dealing with JSON text sequences (rfc7464) luckily jq has got your back too with the --seq parameter.

Edit 2: Both the newline separated JSON and the JSON text sequences have one important advantage; they reduce memory requirements down to O(1), meaning your total memory requirement is only dependent on your longest line of input, whereas putting the entire input in a single array requires that either your parser can handle late errors (i.e. after the first 100k elements there's a syntax error), which generally isn't the case to my knowledge, or it will have to parse the entire file twice (first validating syntax, then parsing, in the process discarding previous elements, as is the case with jq --stream) which also happens rarely to my knowledge, or it will try to parse the whole input at once and return the result in one step (think of receiving a Python dict which contains the entirety of your say 50G input data plus overhead) which is usually memory backed, hence raising your memory footprint by just about your total data size.

Edit 3: If you hit any obstacles, try using keys_unsorted instead of keys. I haven't tested that myself (I kind of assume my columns were already sorted), however @Kyle Barron reports that this was needed.

Edit 4: As pointed out by youngminz in the comment below the original command fails when working with non-{number,string} values like nested lists. The command has been updated (with a slightly adapted version from the comment, map() – unlike map_values() converts objects to their keys the same as [.[]], making the map more readable). Keys remain unaffected, if you really have complex types as keys (which may not even conform to JSON, but I'm too lazy to look it up right now) you can do the same for the key-related mappings.

Edit 5: Edit 3 is now applied to the body of the text in response to feedback in the comments.

4
  • 6
    I found I needed to use keys_unsorted so that the keys had the same order as the record: stedolan.github.io/jq/manual/#keys,keys_unsorted Feb 13, 2020 at 21:41
  • 1
    .import fails when fields have a newline \n. jq -r converts that back to an unescaped newline.
    – Matt
    Jul 16, 2022 at 8:08
  • 1
    I had to deal with JSON files containing nested lists. In this case, I had to use the tostring together. Otherwise, an error like this occurred: jq: error (at <stdin>:703): array ([1,1,0,0,4]) is not valid in a csv row. Complete command with keys_unsorted and tostring: (head -1 data.json | jq -r 'keys_unsorted | @csv' && jq -r '[.[]|tostring] | @csv' < data.json) > data.csv
    – youngminz
    Oct 9, 2022 at 16:14
  • 1
    I tested keys_unsorted and it is indeed required for some input files
    – sdgfsdh
    May 17 at 15:52
8

sqlitebiter appears to provide a python solution:

A CLI tool to convert CSV/Excel/HTML/JSON/LTSV/Markdown/SQLite/TSV/Google-Sheets to a SQLite database file. http://sqlitebiter.rtfd.io/

docs: http://sqlitebiter.readthedocs.io/en/latest/

project: https://github.com/thombashi/sqlitebiter

  • last update approximately 3 months ago
  • last issue closed approximately 1 month ago, none open
  • noted today, 2018-03-14
1
5

You can use spyql. spyql reads the json files (with 1 json object per line) and generates INSERT statements that you can pipe into sqlite:

$ spyql -Otable=my_table "SELECT json->uri, json->user_agent FROM json TO sql" < sample3.json | sqlite3 my.db

This assumes that you already created an empty table in the sqlite database my.db.

Disclaimer: I am the author of spyql.

2

If (as in the original question) the JSON data comes in the form of JSONLines (that is, one JSON entity per line), and if it is desired to create a table with one of these entities per row, then sqlite3 can be used to import the data by setting .mode line and .separator "\t", e.g. as follows:

create table input (
    raw JSON
);
.mode line
.separator "\t"
.import input.json input

This approach is worth knowing not least because it can easily be adapted to handle cases where the data is not already in JSONLines format. For example, if input.json contains a single very long JSON array, we could use a tool such as or gojq to "splat" it:

.mode line
.separator "\t"
.import "|jq -c .[] input.json" input

Similarly, if input.json contains a single object with many keys, and if it is desired to create a table of corresponding single-key objects:

.mode line 
.separator "\t"
.import "|jq -c 'to_entries[] | {(.key): .value}'" input

If the original data is a single very large JSON array or JSON object, jq's streaming parser could be used to save memory. In this context, it may be worth mentioning two CLI tools with minimal memory requirements: my own jm (based on JSON Machine), and jm.py (based on ijson). E.g., to "splat" each array in a file containing one or more JSON arrays:

.mode line
.separator "\t"
.import "|jm input.json" input

With the JSON data safely in an SQLite table, it is (thanks to SQLite's support for JSON) now quite straightforward to create indices, populate other tables, etc., etc.

—- Thanks to @Andrew re .separator.

4
  • Note: I .mode=line errorred with sqlite 3.41.0 2023-02-21 18:09:37. .mode line worked.
    – Andrew
    Feb 28 at 22:07
  • @Andrew. Fixed. Tx.
    – peak
    Mar 1 at 2:51
  • nice! one other issue I found is that the default separator for .import is |, which means this method has the potential to truncate data and create invalid JSON! e.g. {"name":"hi | there"} will get saved as {"name":"hi and you'll get a Malformed JSON error when you try to json_extract() from the column. The only workaround I found was to specify a separator manually that is "safe" for your data. It must be a single character. For my data \t (tab) was safe. .separator "\t" "\n" should be after the .mode line command when ever calling .import.
    – Andrew
    Mar 1 at 16:37
  • I do want to say, this was a very helpful answer for me! In my case, I already knew the column names, so it was straightforward to create a query that would transform the data.
    – Andrew
    Mar 1 at 16:41
1

To work with a file of newline delimited JSON objects, including \n in the data.

Add a header column name and ensure the JSON is compact (1 line per record).

cat <(echo '"line"') source.json | jq -c '.' > source.fauxcsv

Import the JSON and header as a "csv" into a temporary table with a column separator \t that won't occur in the JSON. Then create the real table via SQLites JSON functions.

sqlite3 file.db \
 -cmd '.separator \t \n' \
 -cmd '.import --schema temp source.fauxcsv temp_json_lines' <<-'EOSQL'
   INSERT into records SELECT 
    json_extract(line, '$.rid'),
    coalesce(json_extract(line, '$.created_at'), strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
    json_extract(line, '$.name')
    FROM temp_json_lines;
EOSQL
1

sqlite-utils, from the same developer of datasette, does that:

sqlite-utils insert --nl db.sqlite tablename data.jsonl
0

Here is the first answer compiled into a deno script:

// just for convenience (pathExists)
import {} from "https://deno.land/x/[email protected]/src/stringUtils.ts";

/**
 * @description
 * convert a json db to csv and then to sqlite
 *
 * @note
 * `sqliteTableConstructor` is a string that is used to create the table, if it is specified the csv file *should not* contain a header row.
 * if it's not specified then the csv file *must* contain a header row so it can be used to infer the column names.
 */
const jsonToSqlite = async (
  {
    jsonDbPath,
    jsonToCsvFn,
    sqliteDbPath,
    sqliteTableConstructor,
    tableName,
  }: {
    jsonDbPath: string;
    sqliteDbPath: string;
    tableName: string;
    sqliteTableConstructor?: string;
    // deno-lint-ignore no-explicit-any
    jsonToCsvFn: (jsonDb: any) => string;
  },
) => {
  // convert it into csv
  const csvDbPath = `${jsonDbPath.replace(".json", "")}.csv`;
  if (csvDbPath.pathExists()) {
    console.log(`${csvDbPath} already exists`);
  } else {
    const db = JSON.parse(await Deno.readTextFile(jsonDbPath));
    const csv = jsonToCsvFn(db);
    await Deno.writeTextFile(csvDbPath, csv);
  }

  // convert it to sqlite
  if (sqliteDbPath.pathExists()) {
    console.log(`${sqliteDbPath} already exists`);
  } else {
    const sqlite3 = new Deno.Command("sqlite3", {
      args: [sqliteDbPath],
      stdin: "piped",
      stderr: "null", // required to make sqlite3 work
    }).spawn();
    await sqlite3.stdin.getWriter().write(
      new TextEncoder().encode(
        ".mode csv\n" +
          (sqliteTableConstructor ? `${sqliteTableConstructor};\n` : "") +
          `.import ${csvDbPath} ${tableName}\n` +
          ".exit\n",
      ),
    );
    await sqlite3.status;
  }
};

Example of usage:

   await jsonToSqlite(
    {
      jsonDbPath: "./static/db/db.json",
      sqliteDbPath: "./static/db/db.sqlite",
      tableName: "radio_table",
      sqliteTableConstructor:
        "CREATE TABLE radio_table(name TEXT, country TEXT, language TEXT, votes INT, url TEXT, favicon TEXT)",
      jsonToCsvFn: (
        db: StationDBType[],
      ) => {
        const sanitize = (str: string) =>
          str.trim().replaceAll("\n", " ").replaceAll(",", " ");
        return db.filter((s) => s.name.trim() && s.url.trim())
          .map(
            (station) => {
              return (
                sanitize(station.name) + "," +
                sanitize(station.country) + "," +
                sanitize(station.language) + "," +
                station.votes + "," +
                sanitize(station.url) + "," +
                sanitize(station.favicon)
              );
            },
          ).join("\n");
      },
    },
  );

Edit1:

  • Importing csv to sqlite by defaults sets all column types to string. In this edit I allow the user to create the table first (via an optional constructor) before importing the csv into it, this way he can specify the exact column types.
  • Improve example

Edit2:

  • Turns out that with deno and sqlite-deno you don't need to use csv as an intermediate or shell out to sqlite, here is an example on how to achieve this:

This next code will create a new sql db from the json one.

import { DB } from "https://deno.land/x/[email protected]/mod.ts";

export interface StationDBType {
  name: string;
  country: string;
  language: string;
  votes: number;
  url: string;
  favicon: string;
}

export const db = new DB(":memory:", {memory: true});
db.query(
  "create TABLE radio_table (name TEXT, country TEXT, language TEXT, votes INT, url TEXT, favicon TEXT)",
);
const jsonDb: StationDBType[] = JSON.parse(
  await Deno.readTextFile("static/db/compressed_db.json"),
);

const sanitize = (s: string) => s.replaceAll('"', "").replaceAll("'", "");
db.query(
  `insert into radio_table values ${
    jsonDb.map((station) =>
      "('" +
      sanitize(station.name) +
      "','" +
      sanitize(station.country) +
      "','" +
      sanitize(station.language) +
      "'," +
      station.votes +
      ",'" +
      sanitize(station.url) +
      "','" +
      sanitize(station.favicon) +
      "')"
    ).join(",")
  }`,
);

Deno.writeFileSync("./db.sqlite", db.serialize());
db.close();
-1

I had a similar problem recently and wanted a really simple way to do this. I wrote a small tool, if you want you could give it a try:

https://github.com/Peter554/jsonsqlquery

cat data.jsonl | jsonsqlquery --create-db data.db

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.