91

I'm considering using SQLite as a production database for a site that would receive perhaps 20 simultaneous users, but with the potential for a peak that could be many multiples of that (since the site would be accessible on the open internet and there's always a possibility that someone will post a link somewhere that could drive many people to the site all at once).

Is SQLite a possibility?

I know it's not an ideal production scenario. I'm only asking if this is within the realm of being a realistic possibility.

1
  • 6
    When you meet the limit of SQLite, then your website is successful enough that you can afford a transition to MySQL without a problem ;) Feb 27, 2021 at 13:21

10 Answers 10

99

SQLite doesn't support any kind of concurrency, so you may have problems running it on a production website. If you're looking for a 'lighter' database, perhaps consider trying a contemporary object-document store like CouchDB.

By all means, continue to develop against SQLite, and you're probably fine to use it initially. If you find your application has more users down the track, you're going to want to transition to Postgres or MySQL however.

The author of SQLite addresses this on the website:

SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites). The amount of web traffic that SQLite can handle depends on how heavily the website uses its database. Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound. SQLite has been demonstrated to work with 10 times that amount of traffic.

The SQLite website (https://www.sqlite.org/) uses SQLite itself, of course, and as of this writing (2015), it handles about 400K to 500K HTTP requests per day, about 15-20% of which are dynamic pages touching the database. Dynamic content uses about 200 SQL statements per webpage. This setup runs on a single VM that shares a physical server with 23 others and yet still keeps the load average below 0.1 most of the time.

So I think the long and short of it is, go for it, and if it's not working well for you, making the transition to an enterprise-class database is fairly trivial anyway. Do take care of your schema, however, and design your database with growth and efficiency in mind.


Here's a thread with some more independent comments around using SQLite for a production web application. It sounds like it has been used with some mixed results.


Edit (2014):

Since this answer was posted, SQLite now features a multi-threaded mode and write ahead logging mode which may influence your evaluation of its suitability for low-medium traffic sites.

Charles Leifer has written a blog post about SQLite's WAL (write ahead logging) feature and some well-considered opinions on appropriate use cases.

12
  • 1
    Interesting quote. Too bad it's not from a more independent source than the author of Sqlite himself. I'd prefer to see some outside verification. May 26, 2009 at 22:50
  • 1
    That slidehost thread was a good discussion, particularly this: "Think of a blog, for example. Because multiple users can comment on a single post, multiple concurrent writes aren't that far-fetched. If your blog does not have commenting capabilities, however, there's likely a small chance that two writes will occur at once (unless you have a fair number of authors). So, think about the possible use cases. If there is indeed a chance that two or more users will be writing to the database at a time then I'd avoid SQLite. For more on locking and concurrency: sqlite.org/lockingv3.html" May 26, 2009 at 23:12
  • 2
    I guess this has changed since your comment was posted, but it's very important to note sqlite.org/threadsafe.html which contradicts your first statement, "SQLite doesn't support any kind of concurrency". Feb 1, 2014 at 15:48
  • 2
    Thanks, but afaik at the time of posting SQLite wasn't multithreaded. I've provided a link to sqlite.org/threadsafe.html and amended the post. Jul 14, 2014 at 23:01
  • 1
    Thanks for updating to include the WAL feature. Your post is a good starting point for considering SQLite for minor distributed applications.
    – Wolf
    Jan 19, 2018 at 10:25
20

The small excerpt from SQLite website says it all.

  • Is the data separated from the application by a network? → choose client/server

  • Many concurrent writers? → choose client/server

  • Big data? → choose client/server

  • Otherwise → choose SQLite!

SQLite "just works" (until it doesn't of course)

2
  • 1
    SQLite "just works". -- This may be a good starting point. Isn't it more interesting to think about what happens if SQLite stops to work, for instance if the traffic grows. Would it also be easy to change the DBMS later?
    – Wolf
    Jan 19, 2018 at 10:34
  • If one uses database-agnostic (using ORM) web-frameworks, such as Django (for Python), for example, then I think it won't be much a problem. What do you think? Now I am developing and maintaining such a service using SQLite and planning to move to PostgreSQL if the load would grow.
    – konstunn
    Jul 24, 2019 at 4:00
8

We often use SQLite for internal databases; The employee directory, our calendar of events, and other intranet services all run on lightweight databases. It would be major overkill to be running these apps at the scale we do on a "real" database like mySQL. This is especially true when you factor in that they're running along side 4 other virtual machines on a single mid-range computer.

At one point we had an outward facing site that ran on an sqlite db for months with only a single reboot required. Obviously, it was very low traffic, but it putted along nicely for what it did.

2
  • That's great to know. Thanks. I'm starting to get comfortable with the idea after reading a lot of positive feedback like this. May 26, 2009 at 23:53
  • @carsonwelsh I only hope you also considered contradictory feedback ;)
    – Wolf
    Jan 19, 2018 at 10:37
5

We have encountered a similar option on an environment with absolutely no writes, and we selected using SQLite.

See my blog post on the subject:

Well, the main assumption which makes this solution theoretically possible is that our SQLite database is totally read-only. Our server code should never change it. This would solve any locking problems, as there are no read locks. We could find nowhere on the internet anyone saying there is a problem in high-throughput reading of SQLite when there are no writes - it could be possible!

3
  • Interesting use case. Your blog post is dating back to October 11, 2014. Would you say it's up-to-date or are some important new missing today?
    – Wolf
    Jan 19, 2018 at 10:42
  • @Wolf - I don't know if it is up-to-date (I am not familiar with WAL for example), but with the assumptions still in place, I think the conclusion is still the same.
    – Uri Agassi
    Jan 19, 2018 at 15:39
  • Wal mode avoid writers to block readers, the real challenge for sqlite is the amount of writers at a time which would be queued to wait an amount of time before getting wal busy error
    – Molem
    Jun 14, 2020 at 15:09
1

I think it would depend mostly on what your read/write ratio will be. If it's mostly reading from the database, you may be okay. Multi-user writing in SQLite can be a problem because of how it locks the database.

5
  • Do you mean if there is an attempt to do two simultaneous writes, it will lock until one completes and then let the other through? That doesn't sound like a problem for the data integrity but more about speed. Given the low traffic volumme, it might be tolerable. May 26, 2009 at 22:38
  • It won't block but return SQLITE_BUSY. It's up to the application to decide how to proceed from there, e.g. wait a while and try again or give up and return an error.
    – laalto
    May 27, 2009 at 6:46
  • 1
    Also with wal mode (write ahead logging) sqlite has become more performant, if anyone is still interested.
    – infocyde
    Nov 11, 2017 at 0:06
  • @infocyde Yes, definitely still interested in this :)
    – Wolf
    Jan 19, 2018 at 10:39
  • I wrote a crappy low traffic web site where I am pretty much the only one updating information and it works fine. It is actually pretty dang fast for reads. I'd architect things so you could with out too much pain swap out the back end db from sqlite to something else if and when you do hit a performance wall with multiple writes going on. Again with wal mode you don't have to be quite as paranoid about this as many believe. I plan to build lots of web apps with sqlite, and the ones that get traction will be promoted to sqlserver or another db down the road.
    – infocyde
    Jan 30, 2018 at 23:44
1

People speak about concurrency problems, but sqlite has a way to cache incoming requests and have them wait for some time. It doesn't timeout immediately.

I've read things about the default timeout setting begin zero, meaning it times out immediately and that's nonsense. Maybe people didn't adjust this setting?

2
  • 1
    Think of concurrency and distributed servers for example having load balancers and having multiple databases. This is where sqlite doesn't work. On its own, its very good as a standalone solution. I use it :)
    – radtek
    Jan 18, 2014 at 17:12
  • Might be somewhat outdated since WAL was introduced
    – Wolf
    Jan 19, 2018 at 10:47
0

Depends on the usage of the site. If most of the time you're just reading data, you can pretty much use anything for a DB and cache the data in the application to achieve good performance.

4
  • That's a big 'if'. What if that's not true? May 26, 2009 at 22:47
  • If it's not true, the "with the potential for a peak that could be many multiples" clause in your question would make the situation risky. We both have our doubts if SQLLite is up to the task, which is why I suggested caching in the first place: To avoid being too dependent on its performance if you get a big increase in hits.
    – Badaro
    May 26, 2009 at 23:17
  • I see. But I'm not crazy about the idea of approximating database functionality in my app. I'd rather leave that work to the database. May 26, 2009 at 23:59
  • You don't need to go that far. I was talking more in the lines of caching the processed data your views need for rendering, or even the HTML output if possible.
    – Badaro
    May 27, 2009 at 0:53
0

I am using it in a very low traffic web server (it is a genomic database) and I don't have any problems. But there are only SELECT statements, no writing to the DB involved.

0

To add to an already brilliant answer: Since you are working with a server-less solution in this case, you can say goodbye to replication, or any sort of horizontal scaling of your db, as well as other advanced options. It also isn't the best choice if you have multiple users updating the same exact chunk of information. If you were to shard the database in the future you would have to migrate the data and move to something else. Also if you have a load balancer and multiple systems involved it would be difficult to maintain data centrality if using sqlite. These are just some of the reasons why it isn't recommended. Its great for smaller projects, and great for development.

0

It seems like with queuing you could also get away with avoiding a lot of the concurrency write problems with SQLite. Instead of writing directly to the sqlite db you would write to a queue that then in turn sequentially writes to the sqlite db in a first in first out mode. Not sure if your application reaches to where you would need this if it would be worth writing or just moving on to client/server DB...but a thought.

1
  • If we're talking about a asp.net web application, a static method could be created with a lock keyword to coordinate each request (thread) to commit transaction one by one; this could avoid the sqlite busy error, but each commit would wait more milliseconds as more users want to write to the db at a time
    – Molem
    Jun 14, 2020 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.