3

I have two payloads and want to merge them into single JSON object (streaming join). At few places people are suggesting to use AttributesToJSON, but as one of the JSON does not have fix set of attributes I guess that would not be possible.

First payload is

{  
   "title":"API-Actions Documentation",
   "title_link":"https://api.slack.com/",
   "author_name":"name",
   "author_link":"http://flickr.com/bobby/",
   "author_icon":"http://flickr.com/icons/bobby.jpg",
   "text":"Optional",
   "image_url":"http://my-website.com/path/to/image.jpg",
   "thumb_url":"http://example.com/path/to/thumb.png",
   "footer":null,
   "pretext":"@name",
   "color":"#7CD197"
}

And second one is,

{  
"fields":[  
  {  
     "title":"Priority",
     "value":"low",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"medium",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"high",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"blocker",
     "short":"true"
  }
 ]
}

I want the output as

{   
"title":"API-Actions Documentation",
"title_link":"https://api.slack.com/",
"author_name":"name",
"author_link":"http://flickr.com/bobby/",
"author_icon":"http://flickr.com/icons/bobby.jpg",
"text":"Optional",
"image_url":"http://my-website.com/path/to/image.jpg",
"thumb_url":"http://example.com/path/to/thumb.png",
"footer":null,
"pretext":"@name",
"color":"#7CD197",
"fields":[  
  {  
     "title":"Priority",
     "value":"low",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"medium",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"high",
     "short":"true"
  },
  {  
     "title":"Priority",
     "value":"blocker",
     "short":"true"
    }
   ]
  }

2 Answers 2

9

Easy! Just use MergeContent and set the following configuration:

Merge Format: Binary Concatenation
Minimum Number of Entries: 2
Delimiter Strategy: Text
Header: [
Footer: ]
Demarcator: ,

(You could use MergeRecord but it is a little buggy for me at least).

Then transfer to JoltTrasnformJSON and set Jolt Transformation DSL to Shift and Jolt Specification to:

{
    "*": {
      "*": "&"
    }
}

This should do the job :)

3
  • MergeContent will not give the desired JSON asked for, it will put each JSON as an element of an array, but the desired JSON shows the "fields" element from the second JSON being added as a field to the first JSON... in my opinion that is a join of data which is different than a merge of data Mar 12, 2019 at 15:22
  • 1
    @BryanBende that's why I've told him to transfer it to JoltTrasnformJSON. The first step is to merge the FlowFiles to act on both of the JSON objects, the second step is to extract them both to one object using JoltTransformJSON. Mar 12, 2019 at 15:24
  • I see, my apologies :) I only quickly read the second part of the answer Mar 12, 2019 at 15:24
2

Generally NiFi is not meant to do traditional streaming joins, but this recent thread on the mailing list can help explain what is possible:

http://apache-nifi-users-list.2361937.n4.nabble.com/join-two-datasets-td7039.html

1
  • It isn't fit for complex joins.. But it is perfect for plain simple merges like this :) Mar 12, 2019 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.