2

I have a cluster on GKE and one of my deployments run tornado web app to receive http requests. This deployment is exposed by a LoadBalancer. I send a simple http request to the LoadBalancer ip, which must run on server side for ~10 minutes. After exactly 5 minutes, I get:

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I tried:

  • Using the communication locally on my computer (both client and server) and haven't got the reset.
  • I made kubectl port-forward directly to the deployment (local client -> kubectl port-forward -> deployment -> server) and haven't got the connection reset. So basically I'm pretty sure it's on the loadbalnacer side.
  • I made a backend config with this configuration:
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-bsc-backendconfig
spec:
  timeoutSec: 3600

and my loadbalancer is configured like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"ports": {"5000":"my-bsc-backendconfig"}'
  creationTimestamp: "2020-12-24T10:08:54Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    name: wesnapp-flask
  name: wesnapp-flask-service
  namespace: default
  resourceVersion: "14652233"
  selfLink: /api/v1/namespaces/default/services/wesnapp-flask-service
  uid: a922e9cb-4702-481f-b1a9-e09df1653ff7
spec:
  clusterIP: 10.64.9.113
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31429
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    name: wesnapp-flask
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: x.x.x.x

Any ideas how to solve this problem? Thanks

2
  • Hello, have you tried to reproduce the same behavior in different environment (for example minikube)? The timeout could be specific to your client <-> server configuration (server setting for timeouts for example).
    – Dawid Kruk
    Dec 24, 2020 at 14:44
  • @DawidKruk Yes I have tried, this situation has nothing to do with my server-client sides. Dec 24, 2020 at 15:19

1 Answer 1

2

Your assumption is right. GCE LoadBalancer kills the connection.
As mentioned in this Google document, there is a Stream idle timeout configured to 300 seconds (5 minutes) and can't be changed. HTTP streams become idle after 5 minutes without activity.

2
  • Going forward, I'd suggest that you take a look at Celery project, in case you're not familiar with it already. It might fit your project architecture.
    – hilsenrat
    Jan 1, 2021 at 21:09
  • Thanks for the tip, I'm already using dask as my task manager. Aug 9 at 19:27

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.