33

My kubernetes PKI expired (API server to be exact) and I can't find a way to renew it. The error I get is

May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922595    8751 server.go:417] Version: v1.14.2
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922784    8751 plugins.go:103] No cloud provider specified.
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922800    8751 server.go:754] Client rotation is on, will bootstrap in background
May 27 08:43:51 node1 kubelet[8751]: E0527 08:43:51.925859    8751 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2019-05-24 13:24:42 +0000 UTC
May 27 08:43:51 node1 kubelet[8751]: F0527 08:43:51.925894    8751 server.go:265] failed to run Kubelet: unable to load bootstrap
kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

The documentation on https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ describes how to renew but it only works if the API server is not expired. I have tried to do a

kubeadm alpha cert renew all

and do a reboot but that just made the entire cluster fail so I did a rollback to a snapshot (my cluster is running on VMware).

The cluster is running and all containers seem to work but I can't access it via kubectl so I can't really deploy or query.

My kubernetes version is 1.14.2.

0

13 Answers 13

75

So the solution was to (first a backup)

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot

then

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

that did the job for me and thanks for your hints :)

7
  • 9
    thanks! i just wanted to add that i also had to delete my etcd .crt and .key files from /etc/kubernetes/pki/etcd/ otherwise, i would get error execution phase certs/etcd-peer: failed to write certificate "etcd-peer": failure loading etcd/peer certificate: failed to load certificate: the certificate has expired
    – yee379
    Commented Dec 20, 2019 at 22:12
  • 1
    I have two masters. The cluster was an HA setup using kubespray. So i am confused that on which master should i do this procedure..should i do it in both masters? Also what should be the value of --apiserver-advertise-address then?
    – AnjK
    Commented Jun 17, 2020 at 12:33
  • This works, but what next to do in other nodes which are not master? It says You must be logged in to server (unauthorized) Commented Jun 27, 2020 at 23:06
  • 3
    You just saved the day. FYI: You might consider replace line 3 with kubeadm init phase certs all --apiserver-advertise-address <public ip> --apiserver-cert-extra-sans=<private ip1>,<private ip2> and run systemctl restart docker && system restart kubelet instead of reboot.
    – wizawu
    Commented Mar 16, 2021 at 11:48
  • 2
    I did this and now I'm getting "Unauthorized" on all nodes. Running kubernetes 1.16.9 with a single master, any ideas. Commented May 25, 2021 at 10:29
9

This topic is also discussed in:


Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":

kubeadm alpha certs check-expiration
  • Automatic certificate renewal:
    • kubeadm renews all the certificates during control plane upgrade.
  • Manual certificate renewal:
    • You can renew your certificates manually at any time with the kubeadm alpha certs renew command.
    • This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.

Overall for Kubernetes v1.14 I find this procedure the most helpful:

1
  • Command "kubeadmin alpha certs renew" helped a lot. After that I also needed one additional step to fix "bootstrap-kubelet.conf does not exist" issue which would not allow kubelet to restart. Run the following command "sudo kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf" as described here github.com/kubernetes/kubernetes/issues/84252
    – kasur
    Commented Apr 9, 2020 at 18:41
6

I was using Kubernetes v15.1 and updated my certificates as explained above, but I still got the same error. The /etc/kubernetes/kubelet.conf was still referring to the expired/old "client-certificate-data".

After some research I found out that kubeadm is not updating the /etc/kubernetes/kubelet.conf file if the certificate-renew was not set to true. So please be aware of a bug of kubeadm below version 1.17 (https://github.com/kubernetes/kubeadm/issues/1753).

kubeadm only upgrades if the cluster upgrade was done with certificate-renewal=true. So I manually had to delete the /etc/kubernetes/kubelet.conf and regenerated it with kubeadm init phase kubeconfig kubelet which finally fixed my problem.

4

Try to do cert renewal via kubeadm init phase certs command.

You can check certs expiration via the following command:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text

First, ensure that you have most recent backup of k8s certificates inventory /etc/kubernetes/pki/*.

Delete apiserver.* and apiserver-kubelet-client.* cert files in /etc/kubernetes/pki/ directory.

Spawn a new certificates via kubeadm init phase certs command:

sudo kubeadm init phase certs apiserver

sudo kubeadm init phase certs apiserver-kubelet-client

Restart kubelet and docker daemons:

sudo systemctl restart docker; sudo systemctl restart kubelet

You can find more related information in the official K8s documentation.

5
  • 2
    Thanks .. I did the above and also created the front-proxy-ca. All certs are renewed in /etc/kubernetes/pki but when restarting the kubelet I still get: Commented May 27, 2019 at 14:21
  • 2
    May 27 16:19:43 node1 kubelet[28167]: E0527 16:19:43.234981 28167 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2019-05-24 13:24:42 +0000 UTC May 27 16:19:43 node1 kubelet[28167]: F0527 16:19:43.235227 28167 server.go:265] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory May 27 16:19:43 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a Commented May 27, 2019 at 14:21
  • 3
    I got it working thanks to you .. I just needed to create all the config files again Commented May 28, 2019 at 3:38
  • 1
    @KimNielsen How did you solved this? I have a cluster setup with 2 masters and i renewed the certificates in one master. Then i rebooted the server, and it gives the same error that you got. All the containers that were up previously are down now.
    – AnjK
    Commented Jun 18, 2020 at 5:31
  • I actually just did exactly like I pasted in the solution .. I only had one master at the time so I don't know if its different with multiple masters Commented Jun 22, 2020 at 6:58
4

This will update all certs under /etc/kubernetes/ssl

kubeadm alpha certs renew all --config=/etc/kubernetes/kubeadm-config.yaml

and do this to restart server commpenont:

kill -s SIGHUP $(pidof kube-apiserver)
kill -s SIGHUP $(pidof kube-controller-manager)
kill -s SIGHUP $(pidof kube-scheduler)
2
[root@nrchbs-slp4115 ~]# kubectl get apiservices |egrep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        125m


[root@nrchbs-slp4115 ~]# kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns         ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   20d
metrics-server   ClusterIP   10.99.2.11   <none>        443/TCP                  125m


[root@nrchbs-slp4115 ~]# kubectl get ep -n kube-system
NAME                      ENDPOINTS                                               AGE
kube-controller-manager   <none>                                                  20d
kube-dns                  10.244.0.5:53,10.244.0.6:53,10.244.0.5:53 + 3 more...   20d
kube-scheduler            <none>                                                  20d
metrics-server            10.244.2.97:443                                         125m
[root@nrchbs-slp4115 ~]#
1
  • 1
    It's always extra credit to include a bit of details or a reference for further studying. Makes your answers more useful
    – mw509
    Commented Mar 16, 2020 at 9:37
2

To help anyone else with Multi-Master setup as I was searching for the answer after the first master has been updated on the second master I did this I found this from another question:

kubeadm only upgrades if the cluster upgrade was done with certificate-renewal=true. So I manually had to delete the /etc/kubernetes/kubelet.conf and regenerated it with kubeadm init phase kubeconfig kubelet which finally fixed my problem.

1

I use a config.yaml to configure the Masters so for me, the answer was:

sudo -i
mkdir -p ~/k8s_backup/etcd
cd /etc/kubernetes/pki/
mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/k8s_backup
cd /etc/kubernetes/pki/etcd
mv {healthcheck-client.crt,healthcheck-client.key,peer.crt,peer.key,server.crt,server.key} ~/k8s_backup/etcd/
kubeadm init phase certs all --ignore-preflight-errors=all --config /etc/kubernetes/config.yaml

cd /etc/kubernetes
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/k8s_backup
kubeadm init phase kubeconfig all --config /etc/kubernetes/config.yaml --ignore-preflight-errors=all

For good measure I reboot

shutdown now -r
2
  • What if i have two masters? Should i do the procedure on both the master machines? Is it enough to use --config /etc/kubernetes/config.yaml instead of --apiserver-advertise-address <IP> ?
    – AnjK
    Commented Jun 17, 2020 at 12:30
  • Actually, after reboot, i was not able to connect to the cluster with an error: " The connection to the server 10.0.2.101:6443 was refused - did you specify the right host or port?". When checked with $docker ps there were no containers related to k8s were up.
    – AnjK
    Commented Jun 17, 2020 at 16:13
1

The best solution is as Kim Nielsen wrote.

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot

With the next command, you can check when the new certifications will be expired:

$ kubeadm alpha certs check-expiration --config=/etc/kubernetes/kubeadm-config.yaml 

or

$ kubeadm certs check-expiration --config=/etc/kubernetes/kubeadm-config.yaml

However, if you have more than one master, you need to copy these files on them.

Log in to the second master and do (backup):

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/

Then log in to the first master (where you created new certificates) and do the next commands (node2 should be changed with IP address of second master machine):

$ rsync /etc/kubernetes/pki/*.crt -e ssh root@node2:/etc/kubernetes/pki/
$ rsync /etc/kubernetes/pki/*.key -e ssh root@node2:/etc/kubernetes/pki/
$ rsync /etc/kubernetes/*.conf -e ssh root@node2:/etc/kubernetes/
1

In my case, I've got same issue after certificate renewal. My cluster was built using Kubespray. My kubelet stopped working and it was saying that I do not have /etc/kubernetes/bootstrap-kubelet.conf file so I looked what this config does.

--bootstrap-kubeconfig string
  | Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.

I understood that this file might not be needed.


A note that I renewed k8s 1.19 with:

kubeadm alpha certs renew apiserver-kubelet-client
kubeadm alpha certs renew apiserver
kubeadm alpha certs renew front-proxy-client

... and that was not sufficient.


The solution was

cp -r /etc/kubernetes /etc/kubernetes.backup

kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf
kubeadm alpha kubeconfig user --client-name system:node:YOUR_MASTER_HOSTNAME_IS_HERE --org system:nodes > /etc/kubernetes/kubelet.conf

kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
0

I am able to solve this issue with below steps:

openssl req -new -key <existing.key> -sub "/CN=system:node:<HOST_NAME>/O=system:nodes" -out new.csr
  • <existing.key> - use it from kubelet.conf
  • <HOST_NAME> - hostname in lower case ( refer expired certificate with command openssl ex: openssl x509 -in old.pem -text -noout)
openssl x509 -req -in new.csr -CA <ca.crt> -CAkey <ca.key> -CAcreateserial -out new.crt -days 365
  • <ca.crt> - from master node
  • <ca.key> - from
  • new.crt - is the renewed certificate replace with expired certificate.

Once replaced , restart kubelet and docker or any other container service.

0
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) >/etc/kubernetes/kubelet.conf

Renew the kubelet.conf helps me solve this issue.

0

On my clusters all three control-plane nodes, where down.

I solved it like this:

cp -a /etc/ /etc-saved

kubeadm certs renew all

rm /etc/kubernetes/kubelet.conf 

kubeadm init phase kubeconfig all

reboot

Doing this on all three nodes solved the issue.

1
  • For us this wasn't enough. Problem was that worker nodes woudn't join and this was due to cluster-info confgimap in kube-public namespace didn't get the new cluster cert. So we recreated the configmap with new cert seen in admin.conf.
    – Rm4n
    Commented Dec 30, 2023 at 11:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.