9

I'm a bit confused on how can I put my log entries directly to elasticsearch (not logstash). So far I found a few appenders (log4j.appender.SocketAppender, log4j.appender.server etc.) that allow to send logs to remote host and also ConversionPattern possibility that seems to allow us to convert logs to "elastic-friendly" format, but this approach looks freaky... or do I mistake? Is this the one way to send logs to elastic?

So far I have a such config:

log4j.rootLogger=DEBUG, server
log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=9200
log4j.appender.server.RemoteHost=localhost
log4j.appender.server.ReconnectionDelay=10000
log4j.appender.server.layout.ConversionPattern={"debug_level":"%p","debug_timestamp":"%d{ISO8601}","debug_thread":"%t","debug_file":"%F", "debug_line":"%L","debug_message":"%m"}%n

But I get an error:

log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)

I can't find any useful example so I can't understand what do I do wrong and how to fix it. Thanks.

3

4 Answers 4

5

I've written this appender here Log4J2 Elastic REST Appender if you want to use it. It has the ability to buffer log events based on time and/or number of events before sending it to Elastic (using the _bulk API so that it sends it all in one go). It has been published to Maven Central so it's pretty straight forward.

3
4

If you'd like to check out something new, my Log4j2 Elasticsearch Appenders will give you async logging in batches with failover.

3
  • Thank you, graylog is absolutely satisfying for me Commented Jun 20, 2018 at 6:36
  • But that one has a vulnerability (see mvnrepository.com/artifact/org.appenders.log4j/… )... Is that going to be fixed sometime? Commented Sep 27, 2022 at 9:42
  • @KiraResari, (easily replaceable) dependency has a vulnerability, not the library itself. Fixed in 1.5.6.
    – rfoltyns
    Commented Oct 5, 2022 at 7:57
1

I found solution that fits my requirements most. It's a graylog . Since it's build based on elasticsearch the usage is familiar so I was able to switch to it immediately.

To use it I added this dependency along with basic log4j2 dependencies:

<dependency>
    <groupId>org.graylog2.log4j2</groupId>
    <artifactId>log4j2-gelf</artifactId>
    <version>1.3.2</version>
</dependency>

and use log4j2.json configuration:

{
  "configuration": {
    "status": "info",
    "name": "LOGGER",
    "packages": "org.graylog2.log4j2",
    "appenders": {
      "GELF": {
        "name": "GELF",
        "server": "log.myapp.com",
        "port": "12201",
        "hostName": "my-awsome-app",
        "JSONLayout": {
          "compact": "false",
          "locationInfo": "true",
          "complete": "true",
          "eventEol": "true",
          "properties": "true",
          "propertiesAsList": "true"
        },
        "ThresholdFilter": {
          "level": "info"
        }
      }
    },
    "loggers": {
      "logger": [
        {
          "name": "io.netty",
          "level": "info",
          "additivity": "false",
          "AppenderRef": {
            "ref": "GELF"
          }
        }        
      ],
      "root": {
        "level": "info",
        "AppenderRef": [
          {
            "ref": "GELF"
          }
        ]
      }
    }
  }
}
0

You can send your logs by HTTP, with HttpAppender, see Log4j Appenders

1
  • I'm sure, person who investigates oportunity to store his app logs with elasticsearch and integrate log4j with it, is aware of such thing as http logging. Inappropriate as an answer to the question. The worst approach to answer the question is something like that: - How to install this thing? - Do not install it or try installing it tomorrow.
    – user11814208
    Commented Nov 28, 2022 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.