9

I have a main syslog server that is receiving syslog from several sources, and I want to send those logs to a Graylog cluster. To help the cluster keep up (on some slow VMs), I need to be able to load balance the messages to Graylog, as sometimes they come in massive chunks from the endpoints (some send 5k logs in bursts every 10 seconds).

I'm trying to use nginx as a load balancer for the syslog messages, but I can't seem to get it to work, and it seems to be because nginx is looking for responses from the Graylog servers. With UDP, it's not going to get a response. At least this is what I think is happening.

The error I'm getting is this:

2016/12/01 11:27:59 [error] 2816#2816: *210325 no live upstreams while connecting 
  to upstream, udp client: 10.0.1.1, server: 0.0.0.0:11016, 
  upstream: "juniper_close_stream_backend", bytes from/to client:932/0, 
  bytes from/to upstream:0/0

As an example of this rule in my nginx.conf, it looks like:

stream {
    server {
        listen 11016 udp;
        proxy_pass juniper_close_stream_backend;
    }
    upstream juniper_close_stream_backend {
        server 10.0.1.2:11016;
        server 10.0.1.3:11016;
        server 10.0.1.4:11016;
    }
}

In this instance, my syslog box is 10.0.1.1, and my downstream Graylog boxes are 10.0.1.[2-4]. I see this error message for all of them.

Any clue on what is happening? When I run tcpdump on the Graylog boxes, I'm seeing the traffic coming from the load balancer, which means it's working. But I think nginx is expecting a response and is giving me an error.

1
  • I might have figured this out a little. I set "proxy_responses 0;" in the 'server' portion of the stream, and the error goes away. This basically tells nginx not to expect a response, which is fine. I'd like to get a health_check running though, but it seems as though that option may only be available in the commercial version. Commented Dec 1, 2016 at 18:09

1 Answer 1

18

So this did seem to be the solution (in my note above).

If using my example from above, you want this to look like:

 stream {
     server {
         listen 11016 udp;
         proxy_pass juniper_close_stream_backend;
         proxy_responses 0;
     }
 }

This tells nginx not to expect a response, which it shouldn't need from UDP. I don't know why their examples don't show this when discussing DNS, which can be entirely UDP driven.

4
  • 3
    There shouldn't be a colon after proxy_responses directive Commented Sep 10, 2018 at 10:48
  • 1
    Is there any way to combine stream and http in one config?
    – Marc
    Commented Oct 29, 2019 at 12:04
  • proxy_responses 0; did the trick for me. Supporting 3-5k connections per second. Thank you.
    – Dbl0McJim
    Commented May 20, 2020 at 22:04
  • Have you had an issue with Graylog showing the nginx server's IP address as the remote IP of the syslog messages? Is it possible to proxy the real client's IP? Commented Apr 10, 2023 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.