Questions tagged [apache-nifi]

Apache NiFi is an easy to use, powerful, and reliable distributed system to transform and distribute data.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
6 views

Does Apache Nifi support .xls

I have read many article even nifi-docs https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-poi-nar/1.5.0/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/ but it says ...
Hakim Chandio's user avatar
0 votes
0 answers
17 views

Jolt Json transformation: Set value by reference identifier (not index)

I am trying to create NIFI Jolt transformation where I want to attach some value from another JSON collection by key from original collection (Assign Prefix from EventCodePrefixes by referenced ...
Dalibor's user avatar
0 votes
0 answers
19 views

How to set value of an attribute as value of 'Maximum Rate' property in Control rate processor in nifi

I have a flow where I am using control rate processor to control the data flow, I will submit tasks in druid using those files, for example let's say I have a total of 10 files. I will first ask my ...
Ram's user avatar
  • 21
0 votes
0 answers
11 views

How to Trim and Update Attribute values for NiFi?

I want to trim all of the whitespace leading and/or tailing the subject. NiFi Processors being used: Get File Extract Text Update Attribute --The issue lies here AttributestoJSON ConvertJSONtoSQL ...
PickleRick's user avatar
0 votes
0 answers
8 views

Is there an inbuilt Monitoring solution for Nifi Processors where sometimes the processor getting hung can be reported

Is there an inbuilt Monitoring solution for Nifi Processors where sometimes the processor getting hung can be reported. Also like for now we are creating some dummy data and passing that through the ...
Tushar Bhatt's user avatar
0 votes
0 answers
13 views

I can't find nifi username and password in the nifi-app file in log directory

Iam trying to use nifi in my localhost, it asks for username and password which is supposed to be autogenerated in logs folder, in nifi-app file. but somehow its not there.... I tried re downloading ...
Himanshu Kumar Vaidhya's user avatar
0 votes
0 answers
15 views

How to trigger Invokehttp processor only when flow files pass to downstream?

I have a workflow where I will be getting some number of flowfiles in a day, the flow files will be invoking a druid API to submit a task. I want to limit the flowfile rate that passes my InvokeHttp ...
Ram's user avatar
  • 21
0 votes
0 answers
6 views

Trigger the processor once after ingesting the two records

I'm fetching some records (e.g., 100 records), and I want to fetch only 2 records. Once those two records get ingested, I only have to fetch another two records. Can you please help me on this issue? ...
PM Ravi Teja's user avatar
0 votes
2 answers
50 views

How to merge two JSON arrays into a single merged array using Jolt in NiFi

I have a requirement in which I need to merge two JSON arrays into one. Currently the JSON is like this: [ { "Id": "123", "test": [ { "count&...
codingprabhu's user avatar
0 votes
1 answer
8 views

Persisting Apache NiFi Templates in Kubernetes with Persistent Volumes

I'm facing an issue with Apache NiFi in a Kubernetes environment where, even though I've mounted a volume to the directory where NiFi templates are stored, the templates are not visible after pod ...
Houssem Salem's user avatar
0 votes
1 answer
9 views

Ni-Fi GenerateTableFetch produces no output if the input table is too big

I have created an Apache Ni-Fi data pipeline that fetches data from a MySQL table and after some data transformation loads the data into a postgreSQL table. The first processor is a GenerateTableFetch ...
Berni Hacker's user avatar
0 votes
1 answer
17 views

How can I use '$' character in JOLT?

I'm trying to do some data transformation using JOLT for Apache nifi but I'm running into some problem. The sample input is: { "id": "1234" } The sample output is: { "$...
these_regrets's user avatar
0 votes
1 answer
21 views

Start a NIFI processor using Python SDK of Apache Nifi

I have a python code which is trying to authenticate to my Nifi server using username and password to start a processor inside it. Below is my code: from flask import Flask import nipyapi import ...
Kiran's user avatar
  • 25
0 votes
0 answers
18 views

Manipulate string in CSV using NiFi processor

I have CSV file which is coming from PowerTop tool, I need to modify this CSV before uploading it to target location. I want to start manipulation from row number 13 which is containing string like ...
Lalit Shukla's user avatar
0 votes
0 answers
20 views

NiFi Rest API: /access/token only works once

System: NiFi 1.19, Java 11, Win 64bit, Auth: Username/password NiFi Rest API: I setup a postman collection for getting the access token. At the first call, it works and NiFi returns a valid JWT-Token (...
Frank Leverenz's user avatar
0 votes
0 answers
15 views

Nifi Merge RecordProcessor Does Not Merge JSON received from Kafka

I use ConsumeKafka processor followed by Merge Records and PutHDFS processors. What I observe is that regardless of Merge Record processor config - a lot of single record flow files are added to HDFS -...
oceansize's user avatar
  • 613
0 votes
0 answers
18 views

Get filesize and filename of flowfiles as flow file contents

I need to fetch the file size and file name of each flow file and append them to flow file content. The flow file is of parquet data and it contains some data which I dont need but I want to get ...
Ram's user avatar
  • 21
0 votes
1 answer
51 views

Upgrading spring boot 3 with Jersey client: ClassCastException: attempting to castjar:file:

I'm upgrading my java spring boot app to spring boot 3. I upgraded org.glassfish.jersey.core:jersey-client from version 2 to 3.1.3, that use jakarta instead of javax package. The problem is that one ...
Nam Nguyen's user avatar
0 votes
0 answers
13 views

Why is Nifi reporting an impossibly large queue size?

We have a Nifi that's currently reporting that there are close to 4TB of messages in a queue. However, the nodes only have 500GB of space on each and none of the drives are full. I initially suspected ...
blockcipher's user avatar
  • 2,144
0 votes
0 answers
22 views

How to load parameters from file?

I'm creating several flows that all use the same parameters. However, I have over 100 values that I need as parameters. Is there anyway to load them from a file? I need this specially because ...
oadaval's user avatar
0 votes
2 answers
51 views

JOLT Transformation with repeating elements for each inner array

I need a JOLT spec to do the below transformation in an Apache NiFi JOLT processor. So there are arrays that are all the same length (in this case 3), and so I want to expand it into 3 records with ...
VinceL's user avatar
  • 299
1 vote
0 answers
9 views

Request apachenifi processor for POST request with form-data

I am trying to POST one request from InvokeHTTP 1.20.0 using nifi processor. But whatever setting I change processor doesnt throw any error but still its not requesting to the server. Here is the ...
Piyush's user avatar
  • 544
1 vote
1 answer
38 views

Jolt: Concat of two strings in array

iam trying to concat the two string from the array and have tried the in jolt. i couldn't get out put: input is: [ { "policy": "Name", "setFrstNm": "John&...
Manimaran Dhandapani's user avatar
0 votes
1 answer
18 views

NiFi load results of mysql query into attribute

I'm doing a simple row count query using an ExecuteSQL processor. I need to route flow based on the value. Currently, in addition to the ExecuteSQL processor, I need to use ConvertAvroToJSON AND ...
Hein du Plessis's user avatar
0 votes
0 answers
15 views

I am trying to running Apache-NiFi on one server and my postgresql is on another server

I am trying to fetch data from postgresql database using NIiFi . I am using Quertdatabase processor . I connected database but there is problem . NiFi running but data not fetched from postgresql. ...
Shalini Singh's user avatar
0 votes
1 answer
32 views

Jolt Spec to create a counter based on the column

I am facing a problem, transforming a JSON . Requirement is create a counter based on the column. Input : [ { "shipment_id": "a60e3760-2f7f-41f3-ac8d-2edd424bcfe4", "...
Thiyagaraj Narayanan's user avatar
0 votes
1 answer
11 views

Jolt for deriving a new columns based on substring function ,if-else and then concatenate both the columns

I Need help in deriving the below output using jolt . Input1: [ { "season": "Fall", "year": "2019" } ] Input2: [ { "season": &...
Mahesh Manchineni's user avatar
0 votes
0 answers
13 views

Create Dynamic Json

Salam All I want to create adynamic json from data in database. my solution in SSMS database as store Procedure but I think it is not best solution the json contain nested json and each part must be ...
Sara Ali's user avatar
0 votes
0 answers
25 views

Can I use HashiCorpVaultParameterProvider in NiFi to get key/value secret engine version 2 in Hashicorp Vault?

Currently I need to use HashiCorpVaultParameterProvider to fetch secret from key/value secret engine (version 2) in Vault. But the Nifi document mentioned that it can connect to version 1. So my ...
Nam Nguyen's user avatar
0 votes
0 answers
20 views

Getting a New Access Token for NiFi REST API

I am working with the NiFi REST API and have run into an issue with obtaining access tokens. If I issue a request to the /access/token endpoint, I get an access token back in the response. If I then ...
Jason Richmeier's user avatar
1 vote
1 answer
26 views

NIFI - best way to handle json at performance level

I'm new to using NIFI and I would like t know the best way to make the usecase below. I read a file containing lines in JSON format (but the content is not a valid JSON), retrieve each line and ...
Patrick Baie's user avatar
0 votes
1 answer
33 views

Expect Script working manually not via NiFi Processor

I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in an SFTP ...
scoutjohn13's user avatar
0 votes
1 answer
19 views

apache nifi: 'Execution node' is invalid because processors with incoming connections cannot be scheduled for primary node only

I have a processor in nifi flow. It is based on FetchFile. On scheduling tab, it has Execution set as 'Primary Node'. When I run this template with nifi 1.10.0, it has no issues. But with 1.22.0, I ...
Mandroid's user avatar
  • 6,354
0 votes
3 answers
35 views

NiFi | How to replace text based on length of field in JSON records

I have data coming into NiFi in JSON format. Below is the sample { "company_name": "mycompany", "companytyp": "producer", "Welcomemes": "...
Aditya's user avatar
  • 3
0 votes
1 answer
30 views

How to read an Excel file in Apache nifi and give the Excel data as input to next processor?

I am trying to read Excel file in Apache nifi and get the content as attribute and need to pass the attribute as input to other processor. For eg. If I am having an Excel like below Column1 Column2 ...
Siva's user avatar
  • 1
3 votes
1 answer
65 views

NiFi integration with Sentry.io

I am trying to catch Exceptions from nifi-app.log. Some of the Exceptions Im not able to catch as they occurs only in top-right corner of the proccessor. And also, I would like to send Sentry ...
Michal Heneš's user avatar
0 votes
0 answers
10 views

NiFi - Sending binary files to Rest API?

I've been trying to send a file that is currently in base64 to an API that only accepts binary data. The process I'm using is the following: I get the file with a processor in a JSON with it's ...
Sinuers's user avatar
  • 53
0 votes
0 answers
8 views

Perform case sensitive search in NiFi search box?

Does NiFi support Case-Sensitive searching within its search box? I'm not talking about creating a search function in a processor. I'm trying to find exact matches within our NiFi flows. e.g. I am ...
edjm's user avatar
  • 4,900
1 vote
0 answers
25 views

How can I process FHIR data in Cloudera CDP environment?

I'm developing a NiFi pipeline in CDP to process FHIR data that are stored in an external DB. Is there any specific tool that I can use in Apache NiFi to read and manipulate FHIR data? Or, as ...
Oppie's user avatar
  • 13
-3 votes
0 answers
23 views

How does Apache NiFi separate data from Amazon Lambda logs by time, compress them into ZIP files by size, and store them in Amazon S3 [closed]

Due to the company's business requirements, Apache NiFi needs to be used in AWS to separate Lambda's logs by time. And compress it into a ZIP file according to its size and store it in S3. How should ...
Erwin's user avatar
  • 51
0 votes
1 answer
25 views

Apche NiFi: Keeping multiple versions of a NAR

Is it possible to keep multiple versions of NARs in lib folder? I came across this, which seems to suggest so. What I want is to have my older flow templates keep working in newer versions of nifi(as ...
Mandroid's user avatar
  • 6,354
0 votes
2 answers
46 views

Jolt concatenate only if my two columns are not Null, default value if it is NULL

I am facing a problem, transforming a JSON . Requirement is to concatenate only if my two columns are not Null, default value if any of one column is NULL Input 1 : [ { "division_Code":...
Thiyagaraj Narayanan's user avatar
0 votes
0 answers
14 views

Setup NIFI to integrate with LDAP on a running cluster

I need to configure my secure NIFI cluster to integrate with LDAP. Which procedure is recommended when doing that on a running cluster? Do I need to shut down all the nodes? What steps do I need to ...
EdiM's user avatar
  • 5
0 votes
0 answers
41 views

Indexing irregular String Data with special characters(;+) into Solr using Apache Nifi

Im new to Apache Nifi, im try to implement it as a replacement for Solr Native Data Import Handler(DIH) which is now deprecated in Solr vr.9. Concrete here is the problem: I can import all other ...
emajar's user avatar
  • 1
1 vote
1 answer
33 views

Apache Nifi docker-compose with Traefik reverse proxy does not work with multiple networks

I currently have an Apache NiFi setup that I access via server:port. I now want to route the UI over my traefik reverse proxy, but doing this via docker network and tags does not work as expected: ...
Felix Steiner's user avatar
0 votes
1 answer
25 views

Reading content in plain text file and if the content is pass then it should go to next processor in Apache nifi

I am trying to create a pipeline where a folder has one output text file of python script. If output text file content has 'Pass' word, pipeline should move further. If not it should terminate. How to ...
Siva's user avatar
  • 1
1 vote
2 answers
33 views

Flattening a nested JSON using jolt transform - Very complex

I am facing a problem, transforming a very complex nested JSON using jolt transformation. Input and output detail is given below. Input [ { "metadata": { "event_time": &...
Thiyagaraj Narayanan's user avatar
-1 votes
0 answers
24 views

Using Simplehttpoperator in airflow astronomer to generate access token in nifi

Astro version:9.0 Airflow:2.7 We are migrating all our working dag to astro.In one of the dag, we use simplehttpoperator to generate nifi access token Snippet: t1=SimpleHttpOperator( task_id='...
Yazhini Ravi's user avatar
1 vote
1 answer
25 views

Apache nifi CSVRecordLookupService: 'csv-format' is not a supported property

I have a legacy apache nifi workflow. One of the processors shows an invalid component message: In the properties section of this processor, I see a property 'Lookup Service' mapped to ...
Mandroid's user avatar
  • 6,354
0 votes
0 answers
19 views

Apache Nifi - issues with file listing path filter

I have sub directory YYYYMMDD which is created by system on daily basis and I need to list file using Nifi file listing. So how can I modified path filter to recognise subdirectory (YYYYMMDD) and ...
Sachin kumar Singh's user avatar

1
2 3 4 5
104