5

Recently I have faced a scenario where I needed to execute some sort of DDL and DML migration (I know these acronyms are not applicable for NoSQL, but anyway) on my CosmosDB database. These migration is planned to be automated and executed within CI/CD pipelines.

Apparently its is really hard to find any information online about the functionality I am trying to achieve. Here are some criteria:

  • Store static data in source code repositories. Check. Possible to deploy to DB with some powershell
  • Having javascript stored procedures in source code repos. Check. A bit of powershell again
  • Modify some data to reflect application layer changes. E.g. adding some fields with default values or specific values collection-wide or on selected records. - not check. can't find acceptable solution as of now.

I am really frustrated about last point and trying to avoid creation of app that will apply DML changes using SDK. In .net world AFAIK EntityFramework does not support automated migrations in CosmosDB.

So question to a good community is: is there anything considered a good practice for automating Cosmos DB data modifications?

Does what I have stated above makes any sense or it is another attempt to deal with NoSQL DBMS as with RDBMS?

2 Answers 2

1

While Cosmos doesn't enforce a schema and will accept any new fields or formats you start providing, for example, I have sometimes encountered a scenario where a one-time "migration" of sorts made sense, for example renaming a field. One approach I've used is to create a single-use, on-demand Azure Function with the sole job of reading the existing data, validating, and making needed changes.

This bit of code can be very deliberate about changes and configured with safeguards to make sure it only runs manually/intentionally and is idempotent to be safe to run multiple times with same result. It can be logically separate from your app's codebase but could still used share data access code like repository classes. Unless you schedule downtime, it would need to account for changes that happen during the migration window, which idempotency solves by allowing to run several passes as needed.

Another option to consider is to leverage Azure Data Factory with its integration to read and transform Cosmos data.

1

Yes, stop trying to treat Cosmos DB like SQL Server. There is no DDL for database schema or default values. This is a NoSQL store that doesn't care what you write to the database.

In Cosmos DB, you define your schema in your application's data layer. If you have data model classes defined, add the new properties there with whatever attributes they need and you are done.

3
  • Totally agreed, but still, I add or the logic and some records in my DB must correspond to the changed logic. For instance, if I change enum values which map to collection record field or whatnot. I don't want this migration to be a part of my app. I want it deployed as per my regular CI/CD process - rollout code changes (deploy binaries), rollout DB changes. Schema is not a problem. The dynamic data which have reflection in my model must be changed during deployment.
    – xwrs
    Jul 24, 2020 at 5:54
  • 1
    Then maybe look at creating a separate library or application that handles the versioning of your application and is called from your deployment and include whatever it is you need to update in your database.
    – Mark Brown
    Jul 24, 2020 at 18:11
  • @MarkBrown that's exactly what I was trying to do when I found this question - creating an app/script, to run within the pipeline, that can run some JS stored procs to make sure that config requirements have been populated (similar to a post-deployment script in SQL world). Don't suppose you know where I can find a guide on doing that? 22 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.