0

Source: Azure storage table has three fields: PartitionKey(string), RowKey(string) and Value (string).

Sink: cosmo db container with three fields: id (string), RowKey(string) and Value (object).

I want to convert the source's value to an object instead of a string so that cosmodb indexes it that way. If I do a mapping via the UI it is synced over as a plain string:

{  "Value": "{\"abc\":\"def\",\"id\":\"1a076c19ff8b41489563453ffbbbb931\"}" }

Where I want it to be like:

{ "Value": {"abc":"def", "id":"123"} }

I need to use the dynamic mappings but after reading the documentation it isn't clear how to do this.

4
  • I would just create a copy activity to STG and then run an U-SQL script to transform that data into an object.
    – Kiril1512
    Jul 15, 2019 at 9:22
  • Hi,lucuma.Does my answer helps you?
    – Jay Gong
    Jul 17, 2019 at 1:13
  • If the other answer using adf doesn't pan out I'll look into using a function.
    – lucuma
    Jul 17, 2019 at 13:05
  • @lucuma Hi,any updates now? Does the azure function solution works?
    – Jay Gong
    Jul 24, 2019 at 7:47

2 Answers 2

0

You need to use the data flow ( which is in preview at this time ) , but this is preetty straight forward . The thing which I am not sure if converting the string to a JSON , but there is function which you can use to convert to JSON its called "json" , please do let me know how it goes .

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-create

0

As I know, no such features could help you convert string data to object format in adf cosmos db configuration.

Please consider other ways. Since you are using adf to import data so that you can't use PreTrigger to change the format of created documents.PreTrigger need to be invoked by code or rest api.

So, as workaround, I suggest you using Azure Function Cosmos DB Trigger to process every document when they imported into database. Please refer to my function code:

using System.Collections.Generic;
using Microsoft.Azure.Documents;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json.Linq;
using System;
using Microsoft.Azure.Documents.Client;

namespace TestADF
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([CosmosDBTrigger(
            databaseName: "db",
            collectionName: "item",
            ConnectionStringSetting = "documentdbstring",
            LeaseCollectionName = "leases")]IReadOnlyList<Document> input, TraceWriter log)
        {
            if (input != null && input.Count > 0)
            {
                log.Verbose("Start.........");
                String endpointUrl = "https://***.documents.azure.com:443/";
                String authorizationKey = "key";
                String databaseId = "db";
                String collectionId = "item";

                DocumentClient client = new DocumentClient(new Uri(endpointUrl), authorizationKey);

                for (int i = 0; i < input.Count; i++)
                {
                    Document doc = input[i];
                    if ((doc.GetPropertyValue<String>("Value") == null) || (!doc.GetPropertyValue<String>("Value")))
                    {                       
                        String V= doc.GetPropertyValue<String>("Value");
                        JObject obj = JObject.Parse(V);

                        doc.SetPropertyValue("Value", obj );

                        client.ReplaceDocumentAsync(UriFactory.CreateDocumentUri(databaseId, collectionId, doc.Id), doc);

                        log.Verbose("Update document Id " + doc.Id);
                    }

                }
            }
        }
    }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.