101

Let's say you have 2 very different types of logs such as technical and business logs and you want:

  • raw technical logs be routed towards a graylog2 server using a gelf output,
  • json business logs be stored into an elasticsearch cluster using the dedicated elasticsearch_http output.

I know that with Syslog-NG for instance, the configuration file allow to define several distinct inputs which can then be processed separately before being dispatched; what Logstash seems unable to do. Even if one instance can be initiated with two specific configuration files, all logs take the same channel and are being applied the same processings ...

Should I run as many instances as I have different types of logs?

1
  • 2
    You should accept Ben Lim's correct answer! Sep 19, 2017 at 17:08

3 Answers 3

198

Should I run as many instances as I have different types of logs?

No! You can only run one instance to handle different types of logs.

In the logstash configuration file, you can specific each input with different type. Then in the filter you can use if to distinct different processing, and also at the output you can use "if" output to different destination.

input {
    file {
            type => "technical"
            path => "/home/technical/log"
    }
    file {
            type => "business"
            path => "/home/business/log"
    }
} 
filter {
    if [type] == "technical" {
            # processing .......
    }
    if [type] == "business" {
            # processing .......
    }
}
output {
    if [type] == "technical" {
            # output to gelf
    }
    if [type] == "business" {
            # output to elasticsearch
    }
}

Hope this can help you :)

7
  • 1
    You can also use the type attribute (with the same type => "value" syntax) in the filter and output definitions which should cut down on the extra formatting within the config file a bit. Example: gist.github.com/fairchild/3030472 Per documentation: Add a 'type' field to all events handled by this input. Types are used mainly for filter activation. The type is stored as part of the event itself, so you can also use the type to search for it in the web interface. Mar 28, 2014 at 22:09
  • 5
    Well, it looks like what Ben provided is actually the new way to do this. When I used type => "value" in an output, I got the following message displayed: "You are using a deprecated config setting "type" set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. You can achieve this same behavior with the new conditionals, like: if [type] == "sometype" { stdout { ... } }." I retract my previous comment. :) Mar 29, 2014 at 2:56
  • 1
    Note that the type attribute won't apply if there's already a type field from the input. This is a special attribute that doesn't override and it's documented. I opened a ticket in Elastic and they recommended me to use tags or add_field instead of type
    – BornToCode
    Jun 18, 2017 at 15:05
  • @BornToCode, I don't quite understand. Are you saying there is something wrong with Ben's code? Or that it wouldn't work if the log paths were the same file? What's the scenario where it doesn't work? Sep 19, 2017 at 17:07
  • 2
    @BenLim The OP did not accept your answer but I found it most helpful and up-voted you. Jan 3, 2019 at 16:15
17

I used tags for multiple file input:

input {
    file {
        type => "java"
        path => "/usr/aaa/logs/stdout.log"
        codec => multiline {
            ...
        },
        tags => ["aaa"]
    }

    file {
        type => "java"
        path => "/usr/bbb/logs/stdout.log"
        codec => multiline {
                ...
        }
        tags => ["bbb"]
    }
}
output {
    stdout {
        codec => rubydebug
    }
    if "aaa" in [tags] {
        elasticsearch {
            hosts => ["192.168.100.211:9200"]
            index => "aaa"
            document_type => "aaa-%{+YYYY.MM.dd}"
        }
    }

    if "bbb" in [tags] {
        elasticsearch {
            hosts => ["192.168.100.211:9200"]
            index => "bbb"
            document_type => "bbb-%{+YYYY.MM.dd}"
        }
    }
}
2
  • 1
    This is better than the accepted answer: it allows multiple filebeat inputs in logstash. In such cases, the "type" property is set to "log" and cannot be modified.
    – Régis B.
    Mar 20, 2018 at 16:07
  • But isn't this overwriting tags with the last tag (bbb)? And then again, in the if statememt, if the tags were an array or single string, then both IF's would work. So logically this is incorrect, but maybe logstash has a different logic inside if's
    – meso_2600
    Oct 28, 2018 at 18:06
0

I think logstash can't read more than 2 files in Input section . try the below

input {
    file {
            type => "technical"
            path => "/home/technical/log"
    }
    file {
            type => "business"
            path => "/home/business/log"
    }
 file {
            type => "business1"
            path => "/home/business/log1"
    }
} 
1
  • Yup, run into same 'issue'
    – meso_2600
    Oct 28, 2018 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.