12

Is it possible/are there tools/ best practices etc to migrate data to a NoSQL format from a relational DB.

I have a JEE6 app making use of Hibernate ORM to persist to MySQL but now we wish to move to NoSQL solution but need to bring the existing data with us

Thanks W

6 Answers 6

13

There are some tools to help the migration, but in the end, MySQL is a relational database which has a completely different structure from noSQL databases.

In the end, you will almost always have to do these four steps stated in this article (refers to mongoDB, and you didn't specify, but it applies to any):

1. Get to know MongoDB. Download it, read the tutorials, try some toy projects.

2. Think about how to represent your model in its document store.

3. Migrate the data from the database to MongoDB, probably simply by writing a bunch of SELECT * FROM statements against the database and then loading the data into your MongoDB model using the language of your choice.

4. Rewrite your application code to query MongoDB through statements such as insert() or find().

1
  • I want to migrate sqlite3 to pouchDB. I want to know, do I need to manually write each table as document? Or there is any other way of migrating the DB ?
    – Apogee
    Commented Oct 9, 2018 at 10:21
4

To simplify things a bit, an Oracle database would have complete mastery over what’s being stored in the database. Oracle or any RDBMS would do this by maintaining relationships between chunks of data stored in tables.

A Document Based database on the other hand stores mostly the how the data is being consumed in the system. It achieves this using the Key-Value pairs, with keys playing certain pivotal variable within.

It would depend on the complexity of the system being migrated, on Volumes, functionality to be offered etc. Though there are multiple ways to migrate from a RDBMS to a json based database, a standard approach would involve constructing views on the existing SQL DB and migrating in stages.

Of course the fundamental process in such a migration is to start with Schema re-design as first step. In a document structure such as Mongo JSON or BSON, most parent child relationships can be accommodated into a single structure(or document). For example, Person_ID, Car_Ownership_ID can both be combined to produce a single json document that lists all the owners of a car.

A Sound schema design process would keep in certain goals under consideration, while denormalizing the fully normalized database in the RDBMS. As a good second step, most JOINs will need to be combined into interweaved collections for easier access with the exceptions of certain outer joins.

Once the Schema is ready, an ETL process or a substitutive script can be used to extract, transform and load the newer schematic with a replica of data in the RDBMS.

1

Migrate data to NoSQL. In my opinion, MongoDB is a good choice. There is a tool, Tapdata Replicator, can replicate MySQL, Oracle, SQLServer to MongoDB. It lets you set the source database and target database, and then map the data from source to target. You can set the new data model in MongoDB without any codes. For example, there are 20 tables in the source db MySQL, you can design the data schema in MongoDB, integrate into 3 or 5 collections. Also, you can select 1:1 clone.

0

I did the M110JS "MongoDB for Node.js Developers" course from Mongo University https://university.mongodb.com/ and can wholeheartedly recommend it (it is free as well).

It does cover (I think in only one of its weeks) the differences between SQL and NOSQL, like when not to use foreign keys but rather embed documents, risking duplicates etc and how to handle it.

It only starts every so often, so this is only a long-term approach.

0

Some helpful info and links for new visiters: Migration from RDBMS to NoSQL very well possible but I suggest read up on database design for NoSQL. You are dealing with completely different animal. This will not be your typical ETL/1:1 migration from one RDBMS to another. Basically design for efficient reads/writes and scale (therefore think CPU, Memory usage for queries and not to minimize disk space like in normalized SQL world). Design for scale with choosing the right partition key. You data model most likely will be very denormalized or even duplicate copies in multiple collections based on the data access requirements. Once your design is final, you can write this migration with Java SDK for your selected NoSQL DB. Here are a few links to get your started:

https://learn.microsoft.com/en-us/azure/cosmos-db/modeling-data https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-modeling-nosql.html

0

First, figure out the structure that you want to use for your NoSQL documents.

Then, write a little app that pulls the data from your RDBMS and writes JSON objects to a file. There should be one JSON object for each document in the new NoSQL database. In the file, after each JSON object, there should be a new line character. You should make a separate file for each document collection (users, orders, etc). Here's an example file for a Users collection.

users.json

{ "name", "John", "city": "San Francisco" }
{ "name", "Jane", "city": "New York" }

You can then import that file into your document store. For Mongo, you can use mongoimport to do this step. The command that you would use would look something like:

mongoimport -u root --authenticationDatabase admin --host="localhost:27017" --db="my_db" --collection="Users" --numInsertionWorkers 4 --file="users.json"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.