0

I have a Problem with my grpc connections and nginx.

So, i've got these things : a client, a server, nginx. They all run on the same machine.

The client does some network scanning and is supposed to send it's results to the server,

The server is supposed to take the scanning-results and put them into a database. The server is based on ASP.Net - See appsettings.json The client and server use "grpc-dotnet" with "protobuf". Both also use the SSL certificate and generally they both work as expected.

The nginx proxy is supposed to be used as a Reverse-Proxy -> Clients only need to know one Endpoint for all their Requests. Nginx is only configured to pass grpc requests, nothing else.

First of all, almost all grpc requests do work. But they stop working, if the request gets too big. I dont know the exact size, but it must be under 180kb.

(180kb is the size of the scan results.In a test, sending half of these results succeded, where sending all did not. Custom test data (with the same class of course) did show the same results. If they got too big, it stopped working)

Without nginx, there are no errors while sending data from client to server. With nginx however, the client does recieve a HTTP 504 error.

Content of error.log (nginx)

2021/07/27 13:27:28 [error] 21464#7952: *66 upstream timed out (10060: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat) while sending request to upstream, client: 127.0.0.1, server: , request: "POST /UI.Remoting.Discovery.Contract.DiscoveryService/SaveDiscoveryResult HTTP/2.0", upstream: "grpcs://127.0.0.1:30053", host: "localhost:30051"

"Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat" translates to :

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond


nginx.conf - i did try many configurations - they do not solve my problems. The configurations shown below are just to show you, which ones i've tested, but there might be missing some. I've tested them in multiple different arrangements and selectively tested out each configuration in different areas (server, http, location)

worker_processes  auto;

events {
    worker_connections 1024;
}

http {
    include mime.types;
    ssl_certificate     nginx-selfsigned.crt;
    ssl_certificate_key nginx-selfsigned.key;

    server {
        listen 30051 ssl http2;
        client_header_buffer_size 16k;
        client_header_timeout 250s;
        
        location /UI.Remoting.Discovery.Contract.DiscoveryService{
             client_max_body_size 1000m; 
             client_body_buffer_size 48k; 
             client_body_timeout 250s; 
             keepalive_timeout 250s; 
             grpc_buffer_size 4k; 
             grpc_read_timeout 250s; 
             grpc_send_timeout 250s; 
             output_buffers 4 320k; 
             proxy_request_buffering off; 
             proxy_buffering off; 
             proxy_cache_convert_head off; 
             proxy_connect_timeout 250s; 
             proxy_headers_hash_max_size 1024; 
             proxy_socket_keepalive off; 
             proxy_send_timeout 250s; 
             proxy_read_timeout 250s; 
             grpc_pass grpcs://localhost:30053;
        }
    }
}

Server - appsettings.json

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "Kestrel": {
    "Limits": { 
      "MaxRequestBodySize": null // Other settings did not seem to affect anything
    },
    "Endpoints": {
      "HttpsInlineCertFile": {
        "Url": "https://localhost:30053",
        "Protocols": "Http2",
        "Certificate": {
          "Path": "Cert/nginx-selfsigned.pfx",
          "Password": "<insert pw here>"
        }
      }
    }
  }
} 

The DiscoveryService interface

[ServiceContract]
public interface IDiscoveryService
{
  ValueTask<SaveDiscoveryResultResponse> SaveDiscoveryResultAsync(SaveDiscoveryResultRequest request);
}

Re-implementing the SaveDiscoveryResultAsync to a stream did not yield any success

[ServiceContract]
public interface IDiscoveryService
{
  ValueTask<SaveDiscoveryResultResponse> SaveDiscoveryResultAsync(IAsyncEnumerable<SaveDiscoveryResultRequest> request);
}

What configuration am i missing / doing wrong ?

1

1 Answer 1

0

It seems like there actually is a bug in the current nginx versions. I've posted a new ticket in nginx's ticket system. https://trac.nginx.org/nginx/ticket/2229

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.