5,317
questions
0
votes
1
answer
27
views
Efficiently Collecting Data from HTTP API and Inserting into PostgreSQL using Apache NiFi
I’m new to Apache NiFi and need advice on handling a large number of HTTP API requests and data processing.
Problem:
I must collect JSON data from an HTTP API (REST service), extract ten fields from ...
-1
votes
0
answers
28
views
Cannot create PoolableConnectionFactory (Method not supported) with Hive JDBC Driver
I am experiencing an issue with Apache NiFi when trying to connect to a Hive database using the DBCPConnectionPool processor. The connection fails with the error:
Cannot create ...
0
votes
2
answers
41
views
How to add Kafka offset, partition details in the message's metadata section using NiFi
Actual message read from Kafka:
{
"body": {
"metadata": {
"id": "bce16e11"
},
"eventDetails": {
"eventID": "...
1
vote
1
answer
35
views
Extract text from a file in NiFi
I have a simple use case. I have an incoming file having say user name and user address in each line. User name is from char 1 to char 5 and user address is from char 6 to char 15. How to use ...
0
votes
0
answers
18
views
Apache NiFi to ElasticSearch connection is slow
We are trying to build a data pipeline in Apache Nifi which will:
Pull huge data several MySQL Database (in total more than 150 million rows)
Convert it to JSON format (arrays or objects)
Push them ...
0
votes
1
answer
13
views
Merge the content of flow files if same flow file Attributevalues
I want to merge the content of flow files into single flow file if we found any matches in flow file attribute value.
I donot know the exact values of flow files attribute values so I can't use ...
0
votes
1
answer
30
views
append rows to csv file, using apache nifi
using Apache nifi 2.0.0, i don't see an option to append rows to an existing csv file.
My setup:
some generator, creating a json-string with an array of records
SplitJson to create single records
...
0
votes
0
answers
27
views
Unable to begin the Dataflow using Apache MiNiFi
I am new to MiNiFi, and started to look out for quick tutorials on how to work with MiNiFi.
Found this cloudera - article and followed the same.
But my MiNiFi is not generating any data to my NiFi ...
-1
votes
0
answers
17
views
Dealing with apche nifi
guys iam trying to use nifi in order to run python scripts but iam having troubles with handling error the flow i created is to messy but i wanted to be able to stop the flow once it gives 0 as exit ...
0
votes
0
answers
17
views
getmongo nifi date filter issue
I am unable to use date related queries in getmongo , following simple query is working mongo but its not filtering anything in the getmongo and selecting the complete collection in nifi
{"...
0
votes
0
answers
11
views
Nifi pass output of ExecuteProcess to ListFile processor
I have generated a FlowFile using ExecuteProcess (ls command with multiple filters)
I am able to get File names one by one using SplitText
Now output FlowFile contains multiple flowfiles with filename ...
0
votes
1
answer
37
views
NiFi in podman connection refused with GetMongo when using mongodb://localhost:27017/ hosted in local machine
I have a MongoDB server hosted and running in my local machine using port 27017 and I want to create a simple workflow in NiFi which copies the details from a collection to a directory using GetMongo ...
0
votes
1
answer
21
views
Is it possible to make NiFi Processor to run but pass through unchanged flowfile?
I have this particular use case where I need to call sql procedure using ExecuteSQL but instead of returning procedure result, I need to return exact same flowfile that entered this processor.
Is it ...
0
votes
1
answer
17
views
how to find the number of flow files from the input queue in apache nifi
I want to read number of flow files from a given input flow file as an argument, let say if my input queue contains three flow files, in my next processor i would like to read it as an argument
0
votes
0
answers
32
views
How can I install python dependencies for Nifi Python processors without the --no-cache-dir option to not download the packages everytime again
I am writing a Python processor for Nifi and while everything works, the developement process is really slow because even though hot reloading works, Nifi downloads the dependencies every time again.
...
0
votes
1
answer
22
views
Is it possible to mimic LOOP control structure in NiFi?
I have more than hundreds of tables coming from different databases or API. Dependencies exist among some of those data sources, so I have ordered them and design the whole flow as linear mode: I let ...
0
votes
1
answer
16
views
How to specify a reader service PropertyDefinition in a customer Python Processor?
I have created a minimal python example to set a Reader Service Property Definition:
from nifiapi.properties import PropertyDescriptor
from nifiapi.flowfiletransform import FlowFileTransform, ...
0
votes
0
answers
14
views
How to prevent NiFi flow from being blocked in conditional case
I have a flow group, its content likes this:
+->row_count=0-->LogAttr-->terminate
inport-->RouteOnRowCount--|
+->row_count>0-->...
2
votes
1
answer
35
views
Using a JOLT Spec on NiFi, how do I remove arrays that only contain empty strings?
Here is the input JSON that I will put into the JOLT transformation processor in NiFi.
{
"guests" : {
"amy" : "amy",
"theresa" : [ "Chef" ],
...
1
vote
1
answer
50
views
Flowfile is currently penalized in Apache NiFI between ExtractText and PutSQL processors in NiFi
I have a requirement to read say name of users from a file(all in new lines) and insert to Database(oracle).
I am using GetFile->SplitText->ExtractText->PutSql in NiFi. All works fine for ...
0
votes
1
answer
26
views
Split the array inside the json into multiple rows via JOLT
I want to split the array inside the JSON, into multiple rows via JOLT. Here number of values inside the array can be varied.
Input JSON:
[
{
"stationname": "station green",
...
-2
votes
0
answers
26
views
after running for a period, the processor in NiFi will stop processing data, even though the status still shows as running [closed]
Now, the upstream pipeline in NiFi is accumulating a large amount of data, but the downstream processor is not processing it.
enter image description here
the processor config:
enter image description ...
0
votes
1
answer
21
views
Replace Text processer does not change the part which I need
I have a Replace Text processer and I need to change the part of the file content,
Let me give a example, these lines belongs the file that will be processed ;
030 111111 22222 333333 ...
0
votes
0
answers
22
views
How can I use the invokeHttp processor with the Delete method in nifi?
I'm trying to use this processor by getting a token in a previous step, I can get this token, but when performing the invokehttp with the delete method passing this token, an error occurs:
validation ...
0
votes
0
answers
22
views
What is the recommended way to connect to Databricks from Apache NiFi
We have been using Apache NiFi to connect to Databricks. While upgrading to the DBR 14.3 and Nifi to 1.27 we realized that SelectHiveQL and SelectHive3QL processors are deprecated.
What is the ...
0
votes
0
answers
16
views
What is schema for XMLReader in Nifi with field has tag attribute?
I have a XML file has structure like:
<docs>
<doc>
<field name="id">ac8887db-2105-517f-a858-8c268313eb4b</field>
<field name="id_account"&...
0
votes
1
answer
34
views
Convertion of XML file to CSV in NIFI, Doesn't work
I am new to NiFi, Working on conversion of xml content to csv.
While doing conversion of xml file to csv struck with the output in a flowfile.
Output of the flowfile in convertrecord
xmlHeader,Data,...
-1
votes
0
answers
24
views
NiFi SSL handshake exception on openjdk 11
I installed the latest docker image of NiFi in kubernetes and enable the OIDC authentication.
When authentication, I am able to enter my email address to login.microsoft but after that I am getting a ...
-2
votes
0
answers
42
views
Apache Nifi ExecuteSQL Processor sometimes freeze for the exact number of minutes
Using NiFi 2.0.0-M4
I am trying to fetch data from mssql database using ExecuteSQL processor through DBCPConnectionPool, queries are very light and should return couple of records with less than ...
0
votes
1
answer
76
views
Replace the keys into different codes in JSON by JOLT
I have a JSON input as follows, need to replace the machine key with its name and then need to have following output JSON with the required mapping.
distance - "code" : "X2-P1"
...
0
votes
1
answer
42
views
Replace the Id value with name in JSON by JOLT
I have a json input as follows, need to replace the machine key with it's name and then need to have following output json with the required mapping. Could someone help on this.
distance -> "...
0
votes
1
answer
28
views
Jolt transform single json object into array of objects per attribute
I have a simple input json object that I would like to turn into an array of json objects, with the number of objects being equal to the number of attributes and each object inheriting one specific ...
2
votes
1
answer
38
views
How to get elements from array with Jolt Transform and make new attributes with their values in NiFi
I have a JSON:
{
"FILE": "23, 21, 24, 25"
}
I need it to look like this:
[
{
"FILE": {
"First": "23",
"Second": "...
0
votes
0
answers
40
views
Getting SSLHandshakeException With Apache Nifi ConsumeJMS using JNDI Configurations
Hi We are configuring ConsumeJMS Processor using JNDI Controller service and ConsumeJMS Properties.
as Shown here.
and the controller service configurations are as follow.
This works perfectly in ...
1
vote
1
answer
34
views
How to use Expression Language to convert parts of attribute text into upper case
I am trying to update attribute using nifi expression language. I am stuck now.
source attribute value is :
"key1"="xXx","key2"="yyy","key3"="zzz&...
2
votes
2
answers
29
views
how to drop or truncate flow file content after handling it?
I am dealing with ETL tasks using NiFi. I observed that flow files were flowing from begin to end. In the fact, after some processor, I handled the data already. so I want to drop or truncate the ...
0
votes
0
answers
19
views
How to ingest Elasticsearch data and convert it to SQL tables using Apache Nifi?
I'm an intern tasked with finding a workaround for the limitations of the Elasticsearch SQL API. Specifically, I need to create a process that converts data from Elasticsearch into a SQL format using ...
0
votes
0
answers
17
views
How do we run ConsumeJMS Processor with custom keystore / truststore
I have ConsumeJMS Processor in Nifi, which i have setup using JndiJmsConnectionFactoryHandler.
I want to execute this nifi processor with a custom keystore.
like how we do when we run java class as ...
0
votes
1
answer
15
views
How do I get a particular array element's value whose position keeps changing using NiFi JsonPath
How do I get a particular array element's value whose position keeps changing using NiFi JsonPath
Input JSON:
{
"phones": [
{
"type": "H",
"...
1
vote
1
answer
17
views
Ni-Fi Import Template is not showing
I have Ni-Fi 2.0 M4 system installed in my local system. But From my local system, I couldnt able to import the template from another Ni-Fi Server. The image attached for your reference. Is there any ...
0
votes
0
answers
44
views
Using Apache Nifi with Plc4xSourceProcessor to read modbus data
I'm using Apache Nifi 1.22.0 and Plc4xSourceProcessor 0.12.0 to read ModBus data.
Connection string: modbus-tcp://xx.xx.xx.xx:port seems to work fine.
I'm trying to read from 30001 (Read Input ...
1
vote
1
answer
55
views
Apache NiFi 9.0.0-m4 - trying to set a flow attribute from JSON content
I'm trying to get monitoringType out of this content and to set it in the flow attributes for use later in routing to a particular elastic index
[ {
...
"Average_Wait_on_Journal" : 0....
0
votes
3
answers
45
views
how to insert the whole json record(as a single json string) in NiFi flow file into json column of database
I am using NiFi to load data from database A in json record format and want to insert json record into a json column of a table in database B.
A => ExecuteSQLRecord => [jsonRow,...] => [?] =&...
0
votes
0
answers
55
views
how to convert string to integer or numeric using apache nifi processor UpdateRecord
So, my problem now is that I can't insert it into my database using the PutDatabaseRecord processor. It is reporting this error: "13:42:05 BRT ERROR
PutDatabaseRecord[id=fe4d0a37-0190-1000-7115-...
0
votes
0
answers
50
views
Insert JSON into Postgres via NiFi
I'm trying insert record with JSON into Postgres table with column type of jsonb using AVROSchemaRegistry and Processor "PutDatabaseRecord".
Apache NiFi version: nifi-2.0.0-M2-RC4
Postgres ...
0
votes
1
answer
31
views
Jolt transform to remove duplicate values in the array
I need to remove duplicates from the below JSon, if there are duplicates in the closed Array, only one value should be retained.
In the below example, the Closed Array contains PP21 two times, i want ...
0
votes
0
answers
24
views
How to process the data as Chunks using Ni-Fi?
My source dataset is Postgres, where I fetch data using Ni-Fi and load the data into another database using Ni-Fi. I am have the huge amounts of data in the source table, and I need to process them in ...
0
votes
1
answer
54
views
apache nifi 2.0.0-M4 - trying to relocate a json value to be the main body of json
0.4 - i'm trying to make the value of /message become the whole body of the json. I've tried updateattribute, updaterecord, groovyscript. I cannot find a good example of this anywhere.
my snipped json ...
1
vote
1
answer
34
views
Filter JSON keys with wild card
I'm using NiFi for the first time. I have the basics down, but I think I'm making the common error of thinking programmatically instead of set wise.
I'm given the following JSON:
{
"ABC#01&...
1
vote
1
answer
38
views
Convert Flat json to tree with Nested array
I'm trying convert JSON by NiFi JOLTTransformRecord processor to tree with nested array.
But I am having some trouble with converting the flat JSON to Nested JSON. I have looked at examples and didn't ...