13

I am trying to create a template for a Kubernetes cluster having 1 master and 2 worker nodes. I have installed all the pre-req software and have run the kubeadmn init on my master node. But when i try to run the kubeadmn join which i get as an output of the init command i am getting an error.

[discovery] Created cluster-info discovery client, requesting info
from "https://10.31.2.33:6443" [discovery] Requesting info from
"https://10.31.2.33:6443" again to validate TLS against the pinned
public key [discovery] Cluster info signature and contents are valid
and TLS certificate validates against pinned roots, will use API
Server "10.31.2.33:6443" [discovery] Successfully established
connection with API Server "10.31.2.33:6443" [kubelet] Downloading
configuration for the kubelet from the "kubelet-config-1.12" ConfigMap
in the kube-system namespace [kubelet] Writing kubelet configuration
to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet
environment file with flags to file
"/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the
kubelet service [tlsbootstrap] Waiting for the kubelet to perform the
TLS Bootstrap... [patchnode] Uploading the CRI Socket information
"/var/run/dockershim.sock" to the Node API object "<workernode2>" as
an annotation error uploading crisocket: timed out waiting for the
condition```

I have done a swapoff -a before running this on the workdernode2

I was able to run the join once but after that, as a part of a script, I ran the kubeadmn reset followed by init and join few times where this has started showing up.

Not able to figure out what or where I am doing a mistake.

My main intent is to put all the commands in the form of a shell script (on master node) so that it can be run on a cluster to create a network.

2
  • You said that "i ran the kubeadmn reset followed by init and join few times". Could it be that you did not update token, generated by init and tried to join with not valid one?
    – getslaf
    Nov 29, 2018 at 14:12
  • No i ensured i used the updated token which i grabbed for the new init command that i ran. Also i realized that rerunning reset and init multiple times is corrupting the setup. If i try the script am using to create the template on a new machine it seems to goes fine. Nov 30, 2018 at 15:21

4 Answers 4

16

I had the encountered the following issue after node was rebooted:

[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8smaster" as an annotation
[kubelet-check] Initial timeout of 40s passed.
error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition

Steps to get rid of this issue:

  1. Check the hostname again, after reboot it might have changed.

    sudo vi /etc/hostname 
    sudo vi /etc/hosts
    
  2. Perform the following clean-up actions

    Code:

    sudo kubeadm reset
    rm -rf /var/lib/cni/
    sudo rm -rf /var/lib/cni/
    
    systemctl daemon-reload
    
    systemctl restart kubelet
    
    sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
    
  3. Execute the init action with the special tag as below

    Code:

    sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.10.10.2 --ignore-preflight-errors=all    
    

    (where 10.10.10.2 is the IP of master node and 192.168.0.0/16 is the private subnet assigned for Pods)

9
  • in my case master IP is 192.168.137.2 so should I use --pod-network-cidr=192.168.0.0/16 or let me know which one else to use ....note I am running master as Hyper V's Vm Nov 21, 2019 at 9:29
  • I used root@POC-k8s-master:~# sudo kubeadm init --pod-network-cidr=10.10.0.0/16 --apiserver-advertise-address=192.168.137.2 --ignore-preflight-errors=all [init] Using Kubernetes version: v1.16.3 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at kubernetes.io/docs/setup/cri [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.5. Latest validated version: 18.09 Nov 21, 2019 at 9:34
  • Getting core dns pending ....... root@POC-k8s-master:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5644d7b6d9-7xk42 0/1 Pending 0 91s kube-system coredns-5644d7b6d9-mbrlx 0/1 Pending 0 91s kube-system etcd-poc-k8s-master 1/1 Running 0 51s kube-system kube-apiserver-poc-k8s-master 1/1 Running 0 32s Nov 21, 2019 at 9:37
  • ~# kubeadm config images pull [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.16.3 [config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.16.3 [config/images] Pulled k8s.gcr.io/kube-scheduler:v1.16.3 [config/images] Pulled k8s.gcr.io/kube-proxy:v1.16.3 [config/images] Pulled k8s.gcr.io/pause:3.1 [config/images] Pulled k8s.gcr.io/etcd:3.3.15-0 [config/images] Pulled k8s.gcr.io/coredns:1.6.2 root@POC-k8s-master:~# export KUBECONFIG=/etc/kubernetes/admin.conf root@POC-k8s-master:~# sysctl net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-iptables = 1 Nov 21, 2019 at 9:38
  • root@POC-k8s-master:~# kubectl apply -f raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/… clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created Nov 21, 2019 at 9:39
9

I’ve had the same problem on Ubuntu 16.04 amd64, fixed it with these commands:

swapoff -a    # will turn off the swap 
kubeadm reset
systemctl daemon-reload
systemctl restart kubelet
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X  # will reset iptables

Also, look at that issue in kubeadm GitHub kubeadm swap on the related issue where people still report having the problem after turning swap off.

You may also try to add --fail-swap-on=false flag in /etc/default/kubelet file, it didn't help in my case though.

It seems to be fixed in the latest k8 version because after upgrading the cluster, I haven't experienced the issue.

1
  • I skipped turning off the swap and the systemctl daemon-reload bit as there is a long running job on the server and I didn't want to restart anything. Seems to work without it.
    – CpILL
    Aug 19, 2019 at 0:02
0

I would recommend caution in altering iptables. This after enormous amount of troubleshooting on a 2 node Ubuntu 20.04 kubeadm cluster was down to internal node mapping that the CNI plugin is dependant on.

With sudo hostnamectl set-hostname c23996dd402c.mylabserver.com I changed the master node back to its mapped name and then did a sudo systemctl daemon-reload. I logged out and back in for good measure to ensure the reload worked. I could see the command prompt change back to the default VM node name.

The mapping now matches for hostname to private IP for the cluster:

~$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
# Cloud Server Hostname mapping
172.31.xx.xxx   xxxxxxxx.mylabserver.com
woopsie@master:~$ cat /etc/hostname
xxxxxxxx.mylabserver.com


The worker1 node has the same problem so reverting to system default was the easier way for me. I just changed it sudo hostnamectl set-hostname masterdefaultnodename. IMHO, iptable alterations should be always done after careful consideration and as an escalated measure to fix the issue in my view.

Master done, onto worker1...

kubectl get nodes
NAME                           STATUS     ROLES           AGE   VERSION
masterdefaultnodename.mylabserver.com   Ready      control-plane   33d   v1.25.0
worker1defautnodname.mylabserver.com   NotReady   <none>          33d   v1.25.0

0

some commands provided need to run on all nodes like below.

kubeadm reset
swapoff -a => all nodes.
rm -rf /var/lib/cni/
systemctl restart kubelet
iptables -F
systemctl restart containerd
systemctl daemon-reload

Then we need to run kubeadm init & follow the instructions given in the output.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.