2

Trying to curl the service deployed in k8s cluster from the master node

curl: (7) Failed to connect to localhost port 31796: Connection 
refused

For kubernetes cluster, when I check my iptables on master I get the following .

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             10.100.94.202        /* 
default/some-service: has no endpoints */ tcp dpt:9015 reject-with 
icmp-port-unreachable
REJECT     tcp  --  anywhere             10.103.64.79         /* 
default/some-service: has no endpoints */ tcp dpt:9000 reject-with 
icmp-port-unreachable
REJECT     tcp  --  anywhere             10.107.111.252       /* 
default/some-service: has no endpoints */ tcp dpt:9015 reject-with 
icmp-port-unreachable

if I flush my iptables with

iptables -F

and then curl

curl -v  localhost:31796 

I get the following

* Rebuilt URL to: localhost:31796/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 31796 (#0)
> GET / HTTP/1.1
> Host: localhost:31796
> User-Agent: curl/7.58.0
> Accept: */*    

but soon after it results in

* Rebuilt URL to: localhost:31796/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 31796 failed: Connection refused
* Failed to connect to localhost port 31796: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 31796: Connection 
refused

I'm using the nodePort concept in my service

Details

kubectl get node 

NAME              STATUS   ROLES    AGE   VERSION
ip-Master-IP   Ready    master   26h   v1.12.7
ip-Node1-ip    Ready    <none>   26h   v1.12.7
ip-Node2-ip    Ready    <none>   23h   v1.12.7


Kubectl get pods 
NAME                             READY   STATUS    RESTARTS   AGE
config-service-7dc8fc4ff-5kk88   1/1     Running   0          5h49m

kubectl get svc -o wide

NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          
AGE   SELECTOR
cadmin-server   NodePort    10.109.55.255   <none>        
9015:31796/TCP   22h   app=config-service
kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP          
26h   <none>

Kubectl get cs 

NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"} 

endpoint.yml

apiVersion: v1
kind: Endpoints
metadata:
name: xyz
subsets:
- addresses:
  - ip: node1_ip
  - ip: node2_ip
 ports:
  - port: 31796
  - name: xyz

service.yml
apiVersion: v1
kind: Service
metadata:
name: xyz
namespace: default
annotations:
alb.ingress.kubernetes.io/healthcheck-path: /xyz
labels:
app: xyz
spec:
type: NodePort
ports:
- nodePort: 31796
  port: 8001
  targetPort: 8001
  protocol: TCP
selector:
app: xyz


deployment.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: xyz
name: xyz
namespace: default
spec:

replicas: 1
selector:
matchLabels:
  app: xyz
template:
  metadata:
  labels:
    app:  xyz
  spec:
  containers:
  - name: xyz
    image:  abc
    ports:
     - containerPort: 8001
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 200m
    volumeMounts:
    - mountPath: /app/
      name: config-volume    
    restartPolicy: Always
    imagePullSecrets:
    - name: awslogin


     volumes:
     - configMap:
      name: xyz
      name: config-volume
7
  • 2
    In general you should use NodeIp:NodePort instead of localhost. Please provide more information about your cluster, especially: kubectl get pods,nodes,svc -o wide, kubectl get cs (verify if your cluster, svc and pods are healthy and are running) You should notice Accept policies.
    – Mark
    Oct 4, 2019 at 15:51
  • without information about your nodes IP you should use Node_IP_Nodeport loks like typo 32207 instead of 31796 (without referencing to those mentioned iptables )
    – Mark
    Oct 10, 2019 at 14:34
  • those where private IP so I thought to keep like that, corrected the port typo . I came across one solution that is adding kind : endpoint , where I defined the node IP in addresses . And then I was able to curl with the node ip and the port.But it should have happen without adding it I guess . I have added the endpoint yml above . Oct 11, 2019 at 7:03
  • Please add your yamls for pods and services
    – Mark
    Oct 11, 2019 at 7:44
  • 1
    Du you have any endpoints related to your service kubectl get ep (did you verify your application is running using service/ cluster IP from another pod?)
    – Mark
    Oct 11, 2019 at 14:02

2 Answers 2

1

You can run the following command to check endpoints.

kubectl get endpoints

If endpoint is not showing up for the service. Please check the yml files that you used for creating the loadbalancer and the deployment. Make sure the labels match.

0

As many have pointed out in their comments the Firewall Rule "no endpoints" is inserted by the kubelet service and indicates a broken Service Application Definition or Setup.

# iptables-save
# Generated by iptables-save v1.4.21 on Wed Feb 24 10:10:23 2021
*filter
# [...]
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "default/web-service:http has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 30081 -j REJECT --reject-with icmp-port-unreachable
# [...]

As you have noticed as well the service kubelet constantly monitors the Firewall Rules and inserts or deletes rules dynamically according to the Kubernetes Pod or Service definitions.

# kubectl get service --namespace=default
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP          198d
web-service   NodePort    10.111.188.199   <none>        8201:30081/TCP   194d
# kubectl get pods --namespace=default
No resources found in default namespace.

In this example case a Service is defined but the Pod associated with the Service does not exist.
Still the kube-proxy process listens on the port 30081:

# netstat -lpn | grep -i kube
[...]
tcp   0   0 0.0.0.0:30081   0.0.0.0:*   LISTEN   21542/kube-proxy 
[...]

So the kubelet service inserts a firewall rule to prevent the traffic for the broken service.

Also the kubelet service will delete this rule as soon as you delete the Service definition

# kubectl delete service web-service --namespace=default
service "web-service" deleted
# iptables-save | grep -i "no endpoints" | wc -l
0

As a Side Node:
This rule is also inserted for Kubernetes Definitions that the kubelet Service doesn't like.
As an example your service can have the name "log-service" but can't have the name "web-log".
In the latter case the kubelet Service didn't give a warning but inserted this blocking rule

3
  • When you say the kubelet constantly monitors firewall rules, what exactly do you mean ? The windows kubelet certainly doesn’t do this to my knowledge. Additionally, what firewall rules are you talking about ? There are many different proxy implementations (ipvs, iptables, user space,...). I think possibly your mixing the role of kube-proxy and kubelet here, but am not sure. I do know the kubelet has some integration with iptables, but am not 100% sure what its for ( a quick scan of the kubelet docs yields iptables-bit-drop, mask, and masquerade values...
    – jayunit100
    Jun 19, 2021 at 18:41
  • Having access to the Kubernetes Node you can insert or delete any iptables rules that make the Kubernetes Network. But waiting some some time you will see that inserted rules are deleted and deleted rules are recreated. As in our case at hand you can delete that annoying rule with iptables but already some moments later you will find that kubelet has reinserted it again. Actually the Kubernetes Documentation doesn't say anything about its iptables manipulations but the whole Kubernetes Network is implemented on the base of iptables Rules. Jun 21, 2021 at 18:42
  • As in the case on hand in the system of the Kubernetes Node you won't find any Web Service process listening on any port of the system. Like you would expect a process listening on port 8201. Only the kube-proxy process is actually listening on a port that was especified as NodePort. But studying the iptables rules you will see that sofisticated rules fiddle the traffic into the port 8201 and that way inbound traffic reaches the Web Service process. Jun 21, 2021 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.