Questions tagged [apache-nifi]
Apache NiFi is an easy to use, powerful, and reliable distributed system to transform and distribute data.
5,151
questions
-2
votes
0
answers
6
views
Does Apache Nifi support .xls
I have read many article even nifi-docs
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-poi-nar/1.5.0/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/
but it says ...
0
votes
0
answers
17
views
Jolt Json transformation: Set value by reference identifier (not index)
I am trying to create NIFI Jolt transformation where I want to attach some value from another JSON collection by key from original collection (Assign Prefix from EventCodePrefixes by referenced ...
0
votes
0
answers
19
views
How to set value of an attribute as value of 'Maximum Rate' property in Control rate processor in nifi
I have a flow where I am using control rate processor to control the data flow, I will submit tasks in druid using those files, for example let's say I have a total of 10 files. I will first ask my ...
0
votes
0
answers
11
views
How to Trim and Update Attribute values for NiFi?
I want to trim all of the whitespace leading and/or tailing the subject.
NiFi Processors being used:
Get File
Extract Text
Update Attribute --The issue lies here
AttributestoJSON
ConvertJSONtoSQL
...
0
votes
0
answers
8
views
Is there an inbuilt Monitoring solution for Nifi Processors where sometimes the processor getting hung can be reported
Is there an inbuilt Monitoring solution for Nifi Processors where sometimes the processor getting hung can be reported.
Also like for now we are creating some dummy data and passing that through the ...
0
votes
0
answers
13
views
I can't find nifi username and password in the nifi-app file in log directory
Iam trying to use nifi in my localhost, it asks for username and password which is supposed to be autogenerated in logs folder, in nifi-app file. but somehow its not there....
I tried re downloading ...
0
votes
0
answers
15
views
How to trigger Invokehttp processor only when flow files pass to downstream?
I have a workflow where I will be getting some number of flowfiles in a day, the flow files will be invoking a druid API to submit a task. I want to limit the flowfile rate that passes my InvokeHttp ...
0
votes
0
answers
6
views
Trigger the processor once after ingesting the two records
I'm fetching some records (e.g., 100 records), and I want to fetch only 2 records. Once those two records get ingested, I only have to fetch another two records. Can you please help me on this issue?
...
0
votes
2
answers
50
views
How to merge two JSON arrays into a single merged array using Jolt in NiFi
I have a requirement in which I need to merge two JSON arrays into one. Currently the JSON is like this:
[
{
"Id": "123",
"test": [
{
"count&...
0
votes
1
answer
8
views
Persisting Apache NiFi Templates in Kubernetes with Persistent Volumes
I'm facing an issue with Apache NiFi in a Kubernetes environment where, even though I've mounted a volume to the directory where NiFi templates are stored, the templates are not visible after pod ...
0
votes
1
answer
9
views
Ni-Fi GenerateTableFetch produces no output if the input table is too big
I have created an Apache Ni-Fi data pipeline that fetches data from a MySQL table and after some data transformation loads the data into a postgreSQL table. The first processor is a GenerateTableFetch ...
0
votes
1
answer
17
views
How can I use '$' character in JOLT?
I'm trying to do some data transformation using JOLT for Apache nifi but I'm running into some problem.
The sample input is:
{
"id": "1234"
}
The sample output is:
{
"$...
0
votes
1
answer
21
views
Start a NIFI processor using Python SDK of Apache Nifi
I have a python code which is trying to authenticate to my Nifi server using username and password to start a processor inside it.
Below is my code:
from flask import Flask
import nipyapi
import ...
0
votes
0
answers
18
views
Manipulate string in CSV using NiFi processor
I have CSV file which is coming from PowerTop tool, I need to modify this CSV before uploading it to target location.
I want to start manipulation from row number 13 which is containing string like ...
0
votes
0
answers
20
views
NiFi Rest API: /access/token only works once
System: NiFi 1.19, Java 11, Win 64bit, Auth: Username/password
NiFi Rest API:
I setup a postman collection for getting the access token.
At the first call, it works and NiFi returns a valid JWT-Token (...
0
votes
0
answers
15
views
Nifi Merge RecordProcessor Does Not Merge JSON received from Kafka
I use ConsumeKafka processor followed by Merge Records and PutHDFS processors.
What I observe is that regardless of Merge Record processor config - a lot of single record flow files are added to HDFS -...
0
votes
0
answers
18
views
Get filesize and filename of flowfiles as flow file contents
I need to fetch the file size and file name of each flow file and append them to flow file content.
The flow file is of parquet data and it contains some data which I dont need but I want to get ...
0
votes
1
answer
51
views
Upgrading spring boot 3 with Jersey client: ClassCastException: attempting to castjar:file:
I'm upgrading my java spring boot app to spring boot 3.
I upgraded org.glassfish.jersey.core:jersey-client from version 2 to 3.1.3, that use jakarta instead of javax package.
The problem is that one ...
0
votes
0
answers
13
views
Why is Nifi reporting an impossibly large queue size?
We have a Nifi that's currently reporting that there are close to 4TB of messages in a queue. However, the nodes only have 500GB of space on each and none of the drives are full. I initially suspected ...
0
votes
0
answers
22
views
How to load parameters from file?
I'm creating several flows that all use the same parameters. However, I have over 100 values that I need as parameters.
Is there anyway to load them from a file?
I need this specially because ...
0
votes
2
answers
51
views
JOLT Transformation with repeating elements for each inner array
I need a JOLT spec to do the below transformation in an Apache NiFi JOLT processor. So there are arrays that are all the same length (in this case 3), and so I want to expand it into 3 records with ...
1
vote
0
answers
9
views
Request apachenifi processor for POST request with form-data
I am trying to POST one request from InvokeHTTP 1.20.0 using nifi processor. But whatever setting I change processor doesnt throw any error but still its not requesting to the server. Here is the ...
1
vote
1
answer
38
views
Jolt: Concat of two strings in array
iam trying to concat the two string from the array and have tried the in jolt. i couldn't get out put:
input is:
[
{
"policy": "Name",
"setFrstNm": "John&...
0
votes
1
answer
18
views
NiFi load results of mysql query into attribute
I'm doing a simple row count query using an ExecuteSQL processor. I need to route flow based on the value. Currently, in addition to the ExecuteSQL processor, I need to use ConvertAvroToJSON AND ...
0
votes
0
answers
15
views
I am trying to running Apache-NiFi on one server and my postgresql is on another server
I am trying to fetch data from postgresql database using NIiFi . I am using Quertdatabase processor . I connected database but there is problem . NiFi running but data not fetched from postgresql.
...
0
votes
1
answer
32
views
Jolt Spec to create a counter based on the column
I am facing a problem, transforming a JSON . Requirement is create a counter based on the column.
Input :
[
{
"shipment_id": "a60e3760-2f7f-41f3-ac8d-2edd424bcfe4",
"...
0
votes
1
answer
11
views
Jolt for deriving a new columns based on substring function ,if-else and then concatenate both the columns
I Need help in deriving the below output using jolt .
Input1:
[
{
"season": "Fall",
"year": "2019"
}
]
Input2:
[
{
"season": &...
0
votes
0
answers
13
views
Create Dynamic Json
Salam All
I want to create adynamic json from data in database.
my solution in SSMS database as store Procedure but I think it is not best solution
the json contain nested json and each part must be ...
0
votes
0
answers
25
views
Can I use HashiCorpVaultParameterProvider in NiFi to get key/value secret engine version 2 in Hashicorp Vault?
Currently I need to use HashiCorpVaultParameterProvider to fetch secret from key/value secret engine (version 2) in Vault.
But the Nifi document mentioned that it can connect to version 1.
So my ...
0
votes
0
answers
20
views
Getting a New Access Token for NiFi REST API
I am working with the NiFi REST API and have run into an issue with obtaining access tokens.
If I issue a request to the /access/token endpoint, I get an access token back in the response. If I then ...
1
vote
1
answer
26
views
NIFI - best way to handle json at performance level
I'm new to using NIFI and I would like t know the best way to make the usecase below.
I read a file containing lines in JSON format (but the content is not a valid JSON), retrieve each line and ...
0
votes
1
answer
33
views
Expect Script working manually not via NiFi Processor
I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in an SFTP ...
0
votes
1
answer
19
views
apache nifi: 'Execution node' is invalid because processors with incoming connections cannot be scheduled for primary node only
I have a processor in nifi flow. It is based on FetchFile.
On scheduling tab, it has Execution set as 'Primary Node'.
When I run this template with nifi 1.10.0, it has no issues. But with 1.22.0, I ...
0
votes
3
answers
35
views
NiFi | How to replace text based on length of field in JSON records
I have data coming into NiFi in JSON format. Below is the sample
{
"company_name": "mycompany",
"companytyp": "producer",
"Welcomemes": "...
0
votes
1
answer
30
views
How to read an Excel file in Apache nifi and give the Excel data as input to next processor?
I am trying to read Excel file in Apache nifi and get the content as attribute and need to pass the attribute as input to other processor. For eg. If I am having an Excel like below
Column1 Column2
...
3
votes
1
answer
65
views
NiFi integration with Sentry.io
I am trying to catch Exceptions from nifi-app.log.
Some of the Exceptions Im not able to catch as they occurs only in top-right corner of the proccessor.
And also, I would like to send Sentry ...
0
votes
0
answers
10
views
NiFi - Sending binary files to Rest API?
I've been trying to send a file that is currently in base64 to an API that only accepts binary data.
The process I'm using is the following:
I get the file with a processor in a JSON with it's ...
0
votes
0
answers
8
views
Perform case sensitive search in NiFi search box?
Does NiFi support Case-Sensitive searching within its search box?
I'm not talking about creating a search function in a processor. I'm trying to find exact matches within our NiFi flows.
e.g.
I am ...
1
vote
0
answers
25
views
How can I process FHIR data in Cloudera CDP environment?
I'm developing a NiFi pipeline in CDP to process FHIR data that are stored in an external DB. Is there any specific tool that I can use in Apache NiFi to read and manipulate FHIR data? Or, as ...
-3
votes
0
answers
23
views
How does Apache NiFi separate data from Amazon Lambda logs by time, compress them into ZIP files by size, and store them in Amazon S3 [closed]
Due to the company's business requirements, Apache NiFi needs to be used in AWS to separate Lambda's logs by time.
And compress it into a ZIP file according to its size and store it in S3. How should ...
0
votes
1
answer
25
views
Apche NiFi: Keeping multiple versions of a NAR
Is it possible to keep multiple versions of NARs in lib folder?
I came across this, which seems to suggest so.
What I want is to have my older flow templates keep working in newer versions of nifi(as ...
0
votes
2
answers
46
views
Jolt concatenate only if my two columns are not Null, default value if it is NULL
I am facing a problem, transforming a JSON .
Requirement is to concatenate only if my two columns are not Null, default value if any of one column is NULL
Input 1 :
[
{
"division_Code":...
0
votes
0
answers
14
views
Setup NIFI to integrate with LDAP on a running cluster
I need to configure my secure NIFI cluster to integrate with LDAP.
Which procedure is recommended when doing that on a running cluster?
Do I need to shut down all the nodes?
What steps do I need to ...
0
votes
0
answers
41
views
Indexing irregular String Data with special characters(;+) into Solr using Apache Nifi
Im new to Apache Nifi, im try to implement it as a replacement for Solr Native Data Import Handler(DIH) which is now deprecated in Solr vr.9.
Concrete here is the problem: I can import all other ...
1
vote
1
answer
33
views
Apache Nifi docker-compose with Traefik reverse proxy does not work with multiple networks
I currently have an Apache NiFi setup that I access via server:port. I now want to route the UI over my traefik reverse proxy, but doing this via docker network and tags does not work as expected:
...
0
votes
1
answer
25
views
Reading content in plain text file and if the content is pass then it should go to next processor in Apache nifi
I am trying to create a pipeline where a folder has one output text file of python script. If output text file content has 'Pass' word, pipeline should move further. If not it should terminate. How to ...
1
vote
2
answers
33
views
Flattening a nested JSON using jolt transform - Very complex
I am facing a problem, transforming a very complex nested JSON using jolt transformation. Input and output detail is given below.
Input
[
{
"metadata": {
"event_time": &...
-1
votes
0
answers
24
views
Using Simplehttpoperator in airflow astronomer to generate access token in nifi
Astro version:9.0
Airflow:2.7
We are migrating all our working dag to astro.In one of the dag, we use simplehttpoperator to generate nifi access token
Snippet:
t1=SimpleHttpOperator(
task_id='...
1
vote
1
answer
25
views
Apache nifi CSVRecordLookupService: 'csv-format' is not a supported property
I have a legacy apache nifi workflow. One of the processors shows an invalid component message:
In the properties section of this processor, I see a property 'Lookup Service' mapped to ...
0
votes
0
answers
19
views
Apache Nifi - issues with file listing path filter
I have sub directory YYYYMMDD which is created by system on daily basis and I need to list file using Nifi file listing. So how can I modified path filter to recognise subdirectory (YYYYMMDD) and ...