0

I have a problem parsing date from csv and I cannot find the problem with (one would assume) simple date - dd/MM/yy. Here's structure of my csv file:

Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%
04/04/16,prid=DUBAP,Hilton_PostAuth 2,16-0006_Auction_Banners_300x250_cat4,383UKHilton_300x250,300 x 250,59,0,0.00%

and my logstash.config file:

input {
  file {
    path => "/Users/User/*.csv"
    type => "core2"
    start_position => "beginning"    
  }
}
filter {
  csv {
      columns => ["Date","Key-values","Line Item","Creative","Ad unit","Creative size","Ad server impressions","Ad server clicks","Ad server CTR"]
      separator => ","
  }
  date {
    match => ["Date", "dd/MM/YY"]
  }
  mutate {convert => ["Ad server impressions", "float"]}
  mutate {convert => ["Ad server clicks", "float"]}
  mutate {convert => ["Ad server CTR", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "test1"
        workers => 1
    }
    stdout {}
}

I have also tried combinations with date being "dd/MM/yy" with no luck, Date is not indexed as date, and I can select only @timestamp in Kibana..

I think there must be a simple thing I'm just missing but as for this moment I cannot find it..

Cheers!

EDIT 1:

Please find my console output when logstash starts and how data is being processed:

Settings: Default pipeline workers: 4
Pipeline main started
Failed parsing date from field {:field=>"Date", :value=>"Date", :exception=>"Invalid format: \"Date\"", :config_parsers=>"dd/MM/YY", :config_locale=>"default=en_US", :level=>:warn}
2016-05-06T20:32:48.034Z Pawels-MacBook-Air.local Date,Key-values,Line Item,Creative,Ad unit,Creative size,Ad server impressions,Ad server clicks,Ad server CTR
2016-04-03T23:00:00.000Z Pawels-MacBook-Air.local 04/04/16,prid=DUBAP,Hilton_PostAuth 1,Stop Clicking Around - 300x250,383UKHilton_300x250,300 x 250,31,0,0.00%

It still loads it into Elasticsearch but in Kibana there's no 'Date' field - I can only use @timestamp

Cheers

2
  • Is there any errors in logs? How does date field indexed?
    – alpert
    May 6, 2016 at 20:10
  • Hi, sorry for vague post, please see it updated May 6, 2016 at 20:32

1 Answer 1

6

Actually what date filter does is:

The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.

So with that configuration it reads your date and use it as timestamp field. If you want to use it as a seperate field, configure as:

date {
    match => ["Date", "dd/MM/yy"]
    target => "Date"
}
2
  • Worked like a charm. Could you please edit your answer and remove coma (syntax error) so I can mark it as answered! Thank you so much! Really appreciate it! May 6, 2016 at 22:06
  • Done. I am happy that I could help :)
    – alpert
    May 6, 2016 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.