I am trying to get logs from ClouldTrail into ElasticSearch so that we can see what is going on in our AWS account better.

I have set up both Logstash and ElasticSearch on my machine (Ubuntu 14.04), and can push text from stdin to ElasticSearch. However when I try to use the S3 input nothing is added to ElasticSearch.

Here is the conf file Im using, I have removed my amazon keys

input {
  s3 {
    bucket => 'ko-cloudtrail-log-bucket'
    secret_access_key => ''
    access_key_id => ''
    delete => false
    interval => '60'
    region => 'eu-west-1'
    type => 'CloudTrail'
    codec => cloudtrail {}
    }
}

output {
    stdout {}
    elasticsearch {
            host => '127.0.0.1'
     }
}

I have install the logstash-codec-cloudtrail codec but the documentation is pretty sparse.

I get no errors in my terminal even when running Logstash with -v and nothing is printed to stdout. Is there something I am missing?

Here's my cloudtrail input. It works great with one minor issue- it duplicates records. As the prefix indicates, I put the cloudtrail logs at s3://bucketname/cloudtrail, not the root.

The mutations are optional. The eventSource mutation is to make the logs a little more readable, and the ruby ingest_time gives me a date the record showed up in ELK- otherwise, it only has the time of the event. Finally, I drop a very common record that just adds noise to my system.

input {
  s3 {
    bucket => "bucketname"
    delete => false
    interval => 60 # seconds
    prefix => "cloudtrail/"
    type => "cloudtrail"
    codec => "cloudtrail"
    credentials => "/etc/logstash/s3_credentials.ini"
    sincedb_path => "/opt/logstash_cloudtrail/sincedb"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
      add_field => {
        "document_id" => "%{eventID}"
      }
    }
    if ! [ingest_time] {
      ruby {
        code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
      }
    }


    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
      drop {}
    }
  }
}

The credentials.ini format is explained on the s3 input page; it's just this:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
  • Thank you for the answer, but this doesn't work for me either. What version of logstash are you using as I am using 1.4.2 – user1810626 Apr 27 '15 at 12:15
  • I'm using 1.4.2. – tedder42 Apr 27 '15 at 14:46

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.