Skip to main content
Filter by
Sorted by
Tagged with
0 votes
1 answer
27 views

Efficiently Collecting Data from HTTP API and Inserting into PostgreSQL using Apache NiFi

I’m new to Apache NiFi and need advice on handling a large number of HTTP API requests and data processing. Problem: I must collect JSON data from an HTTP API (REST service), extract ten fields from ...
özüm's user avatar
  • 1,323
-1 votes
0 answers
28 views

Cannot create PoolableConnectionFactory (Method not supported) with Hive JDBC Driver

I am experiencing an issue with Apache NiFi when trying to connect to a Hive database using the DBCPConnectionPool processor. The connection fails with the error: Cannot create ...
Dario Villalon's user avatar
0 votes
2 answers
41 views

How to add Kafka offset, partition details in the message's metadata section using NiFi

Actual message read from Kafka: { "body": { "metadata": { "id": "bce16e11" }, "eventDetails": { "eventID": "...
Sam's user avatar
  • 21
1 vote
1 answer
35 views

Extract text from a file in NiFi

I have a simple use case. I have an incoming file having say user name and user address in each line. User name is from char 1 to char 5 and user address is from char 6 to char 15. How to use ...
Alok Kumar's user avatar
0 votes
0 answers
18 views

Apache NiFi to ElasticSearch connection is slow

We are trying to build a data pipeline in Apache Nifi which will: Pull huge data several MySQL Database (in total more than 150 million rows) Convert it to JSON format (arrays or objects) Push them ...
Ali Valizada's user avatar
0 votes
1 answer
13 views

Merge the content of flow files if same flow file Attributevalues

I want to merge the content of flow files into single flow file if we found any matches in flow file attribute value. I donot know the exact values of flow files attribute values so I can't use ...
Vyshnavi's user avatar
0 votes
1 answer
30 views

append rows to csv file, using apache nifi

using Apache nifi 2.0.0, i don't see an option to append rows to an existing csv file. My setup: some generator, creating a json-string with an array of records SplitJson to create single records ...
MartinK's user avatar
  • 43
0 votes
0 answers
27 views

Unable to begin the Dataflow using Apache MiNiFi

I am new to MiNiFi, and started to look out for quick tutorials on how to work with MiNiFi. Found this cloudera - article and followed the same. But my MiNiFi is not generating any data to my NiFi ...
Vishal's user avatar
  • 3
-1 votes
0 answers
17 views

Dealing with apche nifi

guys iam trying to use nifi in order to run python scripts but iam having troubles with handling error the flow i created is to messy but i wanted to be able to stop the flow once it gives 0 as exit ...
basel nabil's user avatar
0 votes
0 answers
17 views

getmongo nifi date filter issue

I am unable to use date related queries in getmongo , following simple query is working mongo but its not filtering anything in the getmongo and selecting the complete collection in nifi {"...
Noman Amjad's user avatar
0 votes
0 answers
11 views

Nifi pass output of ExecuteProcess to ListFile processor

I have generated a FlowFile using ExecuteProcess (ls command with multiple filters) I am able to get File names one by one using SplitText Now output FlowFile contains multiple flowfiles with filename ...
stack dev list's user avatar
0 votes
1 answer
37 views

NiFi in podman connection refused with GetMongo when using mongodb://localhost:27017/ hosted in local machine

I have a MongoDB server hosted and running in my local machine using port 27017 and I want to create a simple workflow in NiFi which copies the details from a collection to a directory using GetMongo ...
TroyUrtiz's user avatar
0 votes
1 answer
21 views

Is it possible to make NiFi Processor to run but pass through unchanged flowfile?

I have this particular use case where I need to call sql procedure using ExecuteSQL but instead of returning procedure result, I need to return exact same flowfile that entered this processor. Is it ...
Damian Graczyk's user avatar
0 votes
1 answer
17 views

how to find the number of flow files from the input queue in apache nifi

I want to read number of flow files from a given input flow file as an argument, let say if my input queue contains three flow files, in my next processor i would like to read it as an argument
Amarnatha Reddy's user avatar
0 votes
0 answers
32 views

How can I install python dependencies for Nifi Python processors without the --no-cache-dir option to not download the packages everytime again

I am writing a Python processor for Nifi and while everything works, the developement process is really slow because even though hot reloading works, Nifi downloads the dependencies every time again. ...
Celdor's user avatar
  • 26
0 votes
1 answer
22 views

Is it possible to mimic LOOP control structure in NiFi?

I have more than hundreds of tables coming from different databases or API. Dependencies exist among some of those data sources, so I have ordered them and design the whole flow as linear mode: I let ...
Willi's user avatar
  • 377
0 votes
1 answer
16 views

How to specify a reader service PropertyDefinition in a customer Python Processor?

I have created a minimal python example to set a Reader Service Property Definition: from nifiapi.properties import PropertyDescriptor from nifiapi.flowfiletransform import FlowFileTransform, ...
Chris Snow's user avatar
  • 24.4k
0 votes
0 answers
14 views

How to prevent NiFi flow from being blocked in conditional case

I have a flow group, its content likes this: +->row_count=0-->LogAttr-->terminate inport-->RouteOnRowCount--| +->row_count>0-->...
Willi's user avatar
  • 377
2 votes
1 answer
35 views

Using a JOLT Spec on NiFi, how do I remove arrays that only contain empty strings?

Here is the input JSON that I will put into the JOLT transformation processor in NiFi. { "guests" : { "amy" : "amy", "theresa" : [ "Chef" ], ...
Ross's user avatar
  • 21
1 vote
1 answer
50 views

Flowfile is currently penalized in Apache NiFI between ExtractText and PutSQL processors in NiFi

I have a requirement to read say name of users from a file(all in new lines) and insert to Database(oracle). I am using GetFile->SplitText->ExtractText->PutSql in NiFi. All works fine for ...
Alok Kumar's user avatar
0 votes
1 answer
26 views

Split the array inside the json into multiple rows via JOLT

I want to split the array inside the JSON, into multiple rows via JOLT. Here number of values inside the array can be varied. Input JSON: [ { "stationname": "station green", ...
casper's user avatar
  • 39
-2 votes
0 answers
26 views

after running for a period, the processor in NiFi will stop processing data, even though the status still shows as running [closed]

Now, the upstream pipeline in NiFi is accumulating a large amount of data, but the downstream processor is not processing it. enter image description here the processor config: enter image description ...
E LE's user avatar
  • 1
0 votes
1 answer
21 views

Replace Text processer does not change the part which I need

I have a Replace Text processer and I need to change the part of the file content, Let me give a example, these lines belongs the file that will be processed ; 030 111111 22222 333333 ...
Sanberk Küçükkelepçe's user avatar
0 votes
0 answers
22 views

How can I use the invokeHttp processor with the Delete method in nifi?

I'm trying to use this processor by getting a token in a previous step, I can get this token, but when performing the invokehttp with the delete method passing this token, an error occurs: validation ...
Malzinoti's user avatar
0 votes
0 answers
22 views

What is the recommended way to connect to Databricks from Apache NiFi

We have been using Apache NiFi to connect to Databricks. While upgrading to the DBR 14.3 and Nifi to 1.27 we realized that SelectHiveQL and SelectHive3QL processors are deprecated. What is the ...
Vijay Kumar's user avatar
  • 2,677
0 votes
0 answers
16 views

What is schema for XMLReader in Nifi with field has tag attribute?

I have a XML file has structure like: <docs> <doc> <field name="id">ac8887db-2105-517f-a858-8c268313eb4b</field> <field name="id_account"&...
Hiền Nguyễn Thị Ngọc's user avatar
0 votes
1 answer
34 views

Convertion of XML file to CSV in NIFI, Doesn't work

I am new to NiFi, Working on conversion of xml content to csv. While doing conversion of xml file to csv struck with the output in a flowfile. Output of the flowfile in convertrecord xmlHeader,Data,...
Viswa1980's user avatar
-1 votes
0 answers
24 views

NiFi SSL handshake exception on openjdk 11

I installed the latest docker image of NiFi in kubernetes and enable the OIDC authentication. When authentication, I am able to enter my email address to login.microsoft but after that I am getting a ...
Elschr 's user avatar
-2 votes
0 answers
42 views

Apache Nifi ExecuteSQL Processor sometimes freeze for the exact number of minutes

Using NiFi 2.0.0-M4 I am trying to fetch data from mssql database using ExecuteSQL processor through DBCPConnectionPool, queries are very light and should return couple of records with less than ...
Damian Graczyk's user avatar
0 votes
1 answer
76 views

Replace the keys into different codes in JSON by JOLT

I have a JSON input as follows, need to replace the machine key with its name and then need to have following output JSON with the required mapping. distance - "code" : "X2-P1" ...
casper's user avatar
  • 39
0 votes
1 answer
42 views

Replace the Id value with name in JSON by JOLT

I have a json input as follows, need to replace the machine key with it's name and then need to have following output json with the required mapping. Could someone help on this. distance -> "...
casper's user avatar
  • 39
0 votes
1 answer
28 views

Jolt transform single json object into array of objects per attribute

I have a simple input json object that I would like to turn into an array of json objects, with the number of objects being equal to the number of attributes and each object inheriting one specific ...
Nick McGurk's user avatar
2 votes
1 answer
38 views

How to get elements from array with Jolt Transform and make new attributes with their values in NiFi

I have a JSON: { "FILE": "23, 21, 24, 25" } I need it to look like this: [ { "FILE": { "First": "23", "Second": "...
Pauline Rumyantseva's user avatar
0 votes
0 answers
40 views

Getting SSLHandshakeException With Apache Nifi ConsumeJMS using JNDI Configurations

Hi We are configuring ConsumeJMS Processor using JNDI Controller service and ConsumeJMS Properties. as Shown here. and the controller service configurations are as follow. This works perfectly in ...
Akash Rai's user avatar
1 vote
1 answer
34 views

How to use Expression Language to convert parts of attribute text into upper case

I am trying to update attribute using nifi expression language. I am stuck now. source attribute value is : "key1"="xXx","key2"="yyy","key3"="zzz&...
Willi's user avatar
  • 377
2 votes
2 answers
29 views

how to drop or truncate flow file content after handling it?

I am dealing with ETL tasks using NiFi. I observed that flow files were flowing from begin to end. In the fact, after some processor, I handled the data already. so I want to drop or truncate the ...
Willi's user avatar
  • 377
0 votes
0 answers
19 views

How to ingest Elasticsearch data and convert it to SQL tables using Apache Nifi?

I'm an intern tasked with finding a workaround for the limitations of the Elasticsearch SQL API. Specifically, I need to create a process that converts data from Elasticsearch into a SQL format using ...
Sara H's user avatar
  • 3
0 votes
0 answers
17 views

How do we run ConsumeJMS Processor with custom keystore / truststore

I have ConsumeJMS Processor in Nifi, which i have setup using JndiJmsConnectionFactoryHandler. I want to execute this nifi processor with a custom keystore. like how we do when we run java class as ...
Akash Rai's user avatar
0 votes
1 answer
15 views

How do I get a particular array element's value whose position keeps changing using NiFi JsonPath

How do I get a particular array element's value whose position keeps changing using NiFi JsonPath Input JSON: { "phones": [ { "type": "H", "...
Sam's user avatar
  • 21
1 vote
1 answer
17 views

Ni-Fi Import Template is not showing

I have Ni-Fi 2.0 M4 system installed in my local system. But From my local system, I couldnt able to import the template from another Ni-Fi Server. The image attached for your reference. Is there any ...
Raja Pandi G R's user avatar
0 votes
0 answers
44 views

Using Apache Nifi with Plc4xSourceProcessor to read modbus data

I'm using Apache Nifi 1.22.0 and Plc4xSourceProcessor 0.12.0 to read ModBus data. Connection string: modbus-tcp://xx.xx.xx.xx:port seems to work fine. I'm trying to read from 30001 (Read Input ...
Roshan Jayalath's user avatar
1 vote
1 answer
55 views

Apache NiFi 9.0.0-m4 - trying to set a flow attribute from JSON content

I'm trying to get monitoringType out of this content and to set it in the flow attributes for use later in routing to a particular elastic index [ { ... "Average_Wait_on_Journal" : 0....
bigbadmouse's user avatar
0 votes
3 answers
45 views

how to insert the whole json record(as a single json string) in NiFi flow file into json column of database

I am using NiFi to load data from database A in json record format and want to insert json record into a json column of a table in database B. A => ExecuteSQLRecord => [jsonRow,...] => [?] =&...
Willi's user avatar
  • 377
0 votes
0 answers
55 views

how to convert string to integer or numeric using apache nifi processor UpdateRecord

So, my problem now is that I can't insert it into my database using the PutDatabaseRecord processor. It is reporting this error: "13:42:05 BRT ERROR PutDatabaseRecord[id=fe4d0a37-0190-1000-7115-...
Brunno Augusto Ribeiro's user avatar
0 votes
0 answers
50 views

Insert JSON into Postgres via NiFi

I'm trying insert record with JSON into Postgres table with column type of jsonb using AVROSchemaRegistry and Processor "PutDatabaseRecord". Apache NiFi version: nifi-2.0.0-M2-RC4 Postgres ...
0 votes
1 answer
31 views

Jolt transform to remove duplicate values in the array

I need to remove duplicates from the below JSon, if there are duplicates in the closed Array, only one value should be retained. In the below example, the Closed Array contains PP21 two times, i want ...
Thiyagaraj Narayanan's user avatar
0 votes
0 answers
24 views

How to process the data as Chunks using Ni-Fi?

My source dataset is Postgres, where I fetch data using Ni-Fi and load the data into another database using Ni-Fi. I am have the huge amounts of data in the source table, and I need to process them in ...
Raja Pandi G R's user avatar
0 votes
1 answer
54 views

apache nifi 2.0.0-M4 - trying to relocate a json value to be the main body of json

0.4 - i'm trying to make the value of /message become the whole body of the json. I've tried updateattribute, updaterecord, groovyscript. I cannot find a good example of this anywhere. my snipped json ...
bigbadmouse's user avatar
1 vote
1 answer
34 views

Filter JSON keys with wild card

I'm using NiFi for the first time. I have the basics down, but I think I'm making the common error of thinking programmatically instead of set wise. I'm given the following JSON: { "ABC#01&...
JimF's user avatar
  • 171
1 vote
1 answer
38 views

Convert Flat json to tree with Nested array

I'm trying convert JSON by NiFi JOLTTransformRecord processor to tree with nested array. But I am having some trouble with converting the flat JSON to Nested JSON. I have looked at examples and didn't ...

1
2 3 4 5
107