I've problem with conversion unix timestamp attribute in my Apache Nifi Template.
FlowFile: the ListenTCP component listens to incoming Squid logs. The UpdateAttribute component then assigns the appropriate schema.name to the appropriate schema. Subsequently the ConvertRecord component, converts files from the format to csv (in the component GrokReader/RecordWriter/CSVRecordSetWriter I have completed the Timestamp Format line with the format MM/dd/yyyy HH:mm:ss). Finally, the PutFile component writes the output files to the disk.
Based on this solution: https://community.hortonworks.com/articles/131320/using-partitionrecord-grokreaderjsonwriter-to-pars.html
This is how the log sent by Squid looks like using a proxy:
1518442283.483 161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ matt DIRECT/199.27.79.73 text/html
This is Grok Expression (in GrokReader):
%{NUMBER:timestamp}\s+%{NUMBER:duration}\s%{IP:client_address}\s%{WORD:cache_result}/%{POSINT:status_code}\s%{NUMBER:bytes}\s%{WORD:request_method}\s%{NOTSPACE:url}\s(%{NOTSPACE:user}|-)\s%{WORD:hierarchy_code}/%{IPORHOST:server}\s%{NOTSPACE:content_type}
This is schema.name = nifi_logs (in AvroSchemaRegistry):
{
"type": "record",
"name": "nifi_logs",
"fields": [
{ "name": "timestamp", "type": "string" },
{ "name": "duration", "type": "string" },
{ "name": "client_address", "type": "string" },
{ "name": "cache_result", "type": "string" },
{ "name": "status_code", "type": "string" },
{ "name": "bytes", "type": "string" },
{ "name": "request_method", "type": "string" },
{ "name": "url", "type": "string" },
{ "name": "user", "type": "string" },
{ "name": "hierarchy_code", "type": "string" },
{ "name": "server", "type": "string" },
{ "name": "content_type", "type": "string" }
]
}
At the output of the program I get a file in which the column fomat should have the format for example MM/dd/yyyy HH:mm:ss:
timestamp,duration,client_address,cache_result,status_code,bytes,request_method,url,user,hierarchy_code,server,content_type
1518442283.483,161,127.0.0.1,TCP_MISS,200,103701,GET,http://www.cnn.com/,matt,DIRECT,199.27.79.73,text/html
I don't know how to do it, despite many attempts, unfortunately it wasn't possible to change the timestamp format.
I tried these solutions: