5

I've seen many similar questions, but nonecan address my particular inquiry:

I want to have my own JSON based database, similar to how MongoDB and firebase have BSON databases', and I figured the simplest way to do that is just to: store some JSON data in a file, load the file when the node.js application starts (into memory of the node.js server) and whenever I Make a search of some sort, just search through the JSON object loaded in the server using .find (if it sall in a big array) and .sort for Object.keys() etc., and then for every change, make a queue (maybe even on another thread) to store the data back to the file, so that 2 entires can't be stored at once (since they're mainly just accessing it in the JSON object stored in memory already, not reading from the file every time).

The problem: If its a huge database, wouldn't that crash the server for storing that much (like maybe 2GB) of JSON data in memory?

how does MongoDB and other nosql databases actually STORE the data? Do they have a separate file for each document / database, and only load into memory what they need? Is there some other way to store database tables / documents into files for later use (and still be able to (almost) instantly access nested JSON objects? Is BSON somehow different in that it can be accessed faster than JSON data? Do they also store the entire database into memory when its loaded?

1 Answer 1

0

If you're looking for a real DB with all the operations that a DBMS provides, then I'd recommend going with a full-fledged DB like Mongo.

If you're looking for a way to organize your JSON data using some sort of file based storage, you can do that using the fs module. It comes with a function called createReadStream that can let you read a large file using a buffer i.e. not loading the entire file in the memory. There's a similar function called createWriteStream for writing data to a large file without running into memory issues.

However, if you're able to divide your large file into multiple small files, then it's really simple to handle reads and writes without using streams. I've done it myself i.e. read and then sort/modify data in JSON file and write it using the fs package. The best part of this approach is that you don't need to do anything for reading. All you need is just to require the file and it's readily available as JSON that you can parse just like any other JSON object.

5
  • Hmm that's a really good idea to just break up the JSON files and require them, although how exactly should one go about doing that? Break up every database, every table? What if one has nested objects in a particular table entry which themselves are even more numberous than all of the other table entries, then one would run into the same problem of too much memory.. do you happen to know HOW databases like Mongo work ? I would like to theoretically try to re-create something like it, its not just magic, there must be some kind of tecnique that can be duplicated? Apr 1, 2019 at 2:35
  • Mongo has a bit of a complex way of doing things. Honestly, you'll be better of going with Mongo than re-creating something similar.
    – VPaul
    Apr 1, 2019 at 3:05
  • but I like making things from scratch :) DO you know of any kind of article or anywhere or any kind of technique to start from? To me its just a big mystery, do you know anything about at least the way they do things? Apr 1, 2019 at 4:57
  • Even I'm interested in exactly what you are. I too want to develop something like firebase and mongodb. Would really appreciate if someone could post an answer to your question. I'm stuck too XD Mar 24, 2021 at 10:08
  • I also build my own datastore (calling it a datastore, because its not really a database). I also added versioning, so you can go back to previous "version" - and it only saves the changes; You can find it on my gist: gist.github.com/erdesigns-eu/3f25506b15f6e6884143ab2735106899
    – eXXecutor
    Aug 25, 2022 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.