0

I am trying to use ValidateCSV processor in Nifi, but I don't know how to define schema. My output(flowfile) is as below:

> PassCountId,CameraId,EventDate,Counter
> 
> 32340,4,2020-10-14 15:26:20.170,4
> 
> 32341,3,2020-10-14 15:26:51.747,4
> 
> 32342,3,2020-10-14 15:26:57.907,6

I tried below schema but it didn't work.

{
  "type": "record",
  "name": "NifiRecord",
  "fields" : [
    {"name": "PassCountId", "type": "bigint"},
    {"name": "CameraId", "type": "int"},
    {"name": "EventDate", "type": "datetime"},
    {"name": "Counter", "type": "int"}
  ]
}

What is the proper way to define a schema?

I already checked documentation. It didn't help unfortunately. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.ValidateCsv/

Thanks.

1 Answer 1

1

@Tyr here is an example of schema

   "type" : "record",

   "namespace" : "nifi",

   "name" : "nifi",

   "fields" : [

      { "name" : "c1" , "type" :  ["null", "string"] },

      { "name" : "c2" , "type" : ["null", "string"] },

      { "name" : "c3" , "type" : ["null", "string"] }

   ]

}

As per the documentation you can feed the schema validation functions:

 : [ParseBigDecimal, ParseBool, ParseChar, ParseDate, ParseDouble, ParseInt, ParseLong, Optional, DMinMax, Equals, ForbidSubStr, LMinMax, NotNull, Null, RequireHashCode, RequireSubStr, Strlen, StrMinMax, StrNotNullOrEmpty, StrRegEx, Unique, UniqueHashCode, IsIncludedIn] 

My recommendation would be to start with a schema of strings, then experiment with ParseBigDecimal for example for your first column. Work in small testable iterations until you have a fully built schema.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.