84

The status of nodes is reported as unknown

"conditions": [
          {
            "type": "Ready",
            "status": "Unknown",
            "lastHeartbeatTime": "2015-11-12T06:03:19Z",
            "lastTransitionTime": "2015-11-12T06:04:03Z",
            "reason": "Kubelet stopped posting node status."
          }

whle kubectl get nodes return a NOTReady status. What does this imply and how to fix this?

6
  • have exactly same problem here :( I was able to delete node in VirtualBox and then kube-up.sh will create a new one. it didn't really help in my situation, but maybe will help you Commented Nov 12, 2015 at 12:51
  • Is there an api to delete the node? I am not sure how the cluster was set up
    – user_mda
    Commented Nov 12, 2015 at 13:06
  • oh, i didn't even ask what kind of setup you have, though it's local vagrant based on virtualbox Commented Nov 12, 2015 at 13:07
  • This is a physical linux vm, any info on how to either create a new node , or restart an existing one?
    – user_mda
    Commented Nov 12, 2015 at 13:08
  • 5
    The only answer is how you delete a node. Everyone who comes to this question is going to be looking for how to restart one. Can we get an answer for that? Commented Oct 31, 2016 at 15:11

7 Answers 7

59

Get nodes

kubectl get nodes

Result:

NAME            STATUS     AGE
192.168.1.157   NotReady   42d
192.168.1.158   Ready      42d
192.168.1.159   Ready      42d

Describe node

Here is a NotReady on the node of 192.168.1.157. Then debugging this notready node, and you can read offical documents - Application Introspection and Debugging.

kubectl describe node 192.168.1.157

Partial Result:

Conditions:
Type          Status          LastHeartbeatTime                       LastTransitionTime                      Reason                  Message
----          ------          -----------------                       ------------------                      ------                  -------
OutOfDisk     Unknown         Sat, 28 Dec 2016 12:56:01 +0000         Sat, 28 Dec 2016 12:56:41 +0000         NodeStatusUnknown       Kubelet stopped posting node status.
Ready         Unknown         Sat, 28 Dec 2016 12:56:01 +0000         Sat, 28 Dec 2016 12:56:41 +0000         NodeStatusUnknown       Kubelet stopped posting node status.

There is a OutOfDisk on my node, then Kubelet stopped posting node status. So, I must free some disk space, using the command of df on my Ubuntu14.04 I can check the details of memory, and using the command of docker rmi image_id/image_name under the role of su I can remove the useless images.

Login in node

Login in 192.168.1.157 by using ssh, like ssh [email protected], and switch to the 'su' by sudo su;

Restart kubelet

/etc/init.d/kubelet restart

Result:

stop: Unknown instance: 
kubelet start/running, process 59261

Get nodes again

On the master:

kubectl get nodes

Result:

NAME            STATUS    AGE
192.168.1.157   Ready     42d
192.168.1.158   Ready     42d
192.168.1.159   Ready     42d

Ok, that node works fine.

Here is a reference: Kubernetes

11

I had an onpremises HA installation, a master and a worker stopped working returning a NOTReady status. Checking the kubelet logs on the nodes I found out this problem:

failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false

Disabling swap on nodes with

swapoff -a

and restarting the kubelet

systemctl restart kubelet

did the work.

1
  • Your issue probably was that swapoff -a only disables the swap temporary until next server restart. To disable to permanently you should remove it from /etc/fstab
    – Tseng
    Commented Aug 11, 2022 at 7:18
8

You can delete the node from the master by issuing:

kubectl delete node hostname.company.net

The NOTReady status probably means that the master can't access the kubelet service. Check if everything is OK on the client.

1
  • 6
    You may have to use following command to delete a node from cluster gracefully. kubectl drain <node-name> --ignore-daemonsets --delete-local-data then kubectl delete node <node-name> Commented Nov 26, 2019 at 5:39
8

If a node is so unhealthy that the master can't get status from it -- Kubernetes may not be able to restart the node. And if health checks aren't working, what hope do you have of accessing the node by SSH?

In this case, you may have to hard-reboot -- or, if your hardware is in the cloud, let your provider do it.

For example, the AWS EC2 Dashboard allows you to right-click an instance to pull up an "Instance State" menu -- from which you can reboot/terminate an unresponsive node.

Before doing this, you might choose to kubectl cordon node for good measure. And you may find kubectl delete node to be an important part of the process for getting things back to normal -- if the node doesn't automatically rejoin the cluster after a reboot.


Why would a node become unresponsive? Probably some resource has been exhausted in a way that prevents the host operating system from handling new requests in a timely manner. This could be disk, or network -- but the more insidious case is out-of-memory (OOM), which Linux handles poorly.

To help Kubernetes manage node memory safely, it's a good idea to do both of the following:

  • Reserve some memory for the system.
  • Be very careful with (avoid) opportunistic memory specifications for your pods. In other words, don't allow different values of requests and limits for memory.

The idea here is to avoid the complications associated with memory overcommit, because memory is incompressible, and both Linux and Kubernetes' OOM killers may not trigger before the node has already become unhealthy and unreachable.

2
  • Thanks for the detailed explanation. In short, if you are using aws ec2 nodes, go to the console and reboot them and your node status may change from NotReady to Ready if you already solved the causing issues. Worked for me. Commented Nov 12, 2020 at 2:34
  • Very nice explanation. That was the case for me after hammering my websockets server with Artillery. ~5K open connections. I rebooted the instance manually from the AWS console and after ~5mins it became ready again..pheww Commented Mar 10, 2023 at 14:58
6

In my case I am running 3 nodes in VM's by using Hyper-V. By using the following steps I was able to "restart" the cluster after restarting all VM's.

  1. (Optional) Swap off

    $ swapoff -a

  2. You have to restart all Docker containers

    $ docker restart $(docker ps -a -q)

  3. Check the nodes status after you performed step 1 and 2 on all nodes (the status is NotReady)

    $ kubectl get nodes

  4. Restart the node

    $ systemctl restart kubelet

  5. Check again the status (now should be in Ready status)

Note: I do not know if it does metter the order of nodes restarting, but I choose to start with the k8s master node and after with the minions. Also it will take a little bit to change the node state from NotReady to Ready

3

In my case I was using EKS. Just needed to reboot it from the aws console.

1
  • 2
    May I know how you restarted it through aws console. ? I am facing similar issue now Commented Feb 23, 2023 at 0:58
-2

I had this problem too but it looks like it depends on the Kubernetes offering and how everything was installed. In Azure, if you are using acs-engine install, you can find the shell script that is actually being run to provision it at:

/opt/azure/containers/provision.sh

To get a more fine-grained understanding, just read through it and run the commands that it specifies. For me, I had to run as root:

systemctl enable kubectl
systemctl restart kubectl

I don't know if the enable is necessary and I can't say if these will work with your particular installation, but it definitely worked for me.

1
  • Please help me understand how removing/installing the service used to manage the resources within Kubernetes can cause a NODE to restart. This is playing havoc on my mind. Commented Jan 16, 2020 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.