How to Renew Kubeadm Expired Certificates
System-wide kubeadm certificates in Kubernetes
In Kubernetes, the system-wide certificates generated by kubeadm are critical for securing communication between the components of the cluster. They provide mutual authentication, encryption, and integrity among the control-plane components and worker nodes.
When you create a cluster with kubeadm or join nodes to it, certificates are generated automatically and stored on the control-plane node(s). These certificates have a default expiration period of 365 days (one year). If they are not renewed before they expire, the cluster can stop functioning correctly, because the secure communication between components will fail.
How kubeadm certificates are used
Certificates generated by kubeadm follow the TLS (Transport Layer Security) protocol and are stored on the cluster in /etc/kubernetes/pki. They are used to:
1. Secure communication between control-plane components
Control-plane components such as the kube-apiserver, kube-controller-manager, kube-scheduler, and etcd use certificates to communicate securely with one another — for example, the kube-apiserver and etcd require a certificate to encrypt their communication.
2. Authenticate worker nodes to the API server
Worker nodes (via the kubelet) need client authentication certificates to connect securely to the kube-apiserver for tasks such as pod scheduling, retrieving configuration, and updating status.
3. Enable secure connections between users and the cluster
Users (via tools like kubectl) interact with the API server securely using client certificates and keys.
4. Serve Kubernetes webhooks and extensions
Webhooks, admission controllers, and custom extensions often use certificates to interact with the cluster securely.
5. Enable the API server to use TLS
The API server uses certificates to handle TLS for incoming requests from nodes, users, or third-party applications.
Checking and manually renewing certificates
By default, kubeadm certificates expire every 365 days. When they expire, you will see connectivity issues and errors such as:
Unable to connect to the server: x509: certificate has expired or is not yet valid
couldn't get current server API group list: Get "https://10.13.12.114:6443/api?timeout=32s": x509: certificate has expired or is not yet valid
These errors prevent the core controllers from establishing TLS connections with the API server, so address certificate expiration promptly.
Check the certificate expiration
| $ sudo kubeadm certs check-expiration |
Back up the old certificates and configs
| mkdir -p $HOME/k8s-old-certs/pki sudo /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki mkdir -p $HOME/k8s-old-certs/pki/etcd sudo /bin/cp -p /etc/kubernetes/pki/etcd/*.* $HOME/k8s-old-certs/pki/etcd sudo /bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs mkdir -p $HOME/k8s-old-certs/.kube sudo /bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/. |
Renew the certificates
| $ sudo kubeadm certs renew all |
As the command output notes, after renewal you must restart the control-plane components (kube-apiserver, kube-controller-manager, kube-scheduler, and etcd) so they pick up the new certificates.
Renew the kubelet certificate
You will find four files in /var/lib/kubelet/pki/, one of which is kubelet.crt. This file expires as well, which you can confirm with openssl:
| sudo ls /var/lib/kubelet/pki kubelet-client-2023-09-28-23-51-54.pem kubelet-client-2024-08-09-20-31-07.pem kubelet-client-current.pem kubelet.crt kubelet.key sudo cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -noout -enddate notAfter=Sep 27 22:51:54 2024 GMT |
Stop the service and delete the old certificates
| sudo systemctl stop kubelet sudo rm /etc/kubernetes/kubelet.conf sudo ls /var/lib/kubelet/pki sudo rm /var/lib/kubelet/pki/kubelet-client-current.pem sudo rm /var/lib/kubelet/pki/kubelet.crt sudo rm /var/lib/kubelet/pki/kubelet.key |
Restore the kubelet service
| sudo kubeadm init phase kubeconfig kubelet sudo systemctl start kubelet |
Update the client configuration
| sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config |
If certificates have already expired
Note: always try to renew certificates before they expire, to avoid an unplanned interruption of operations.
If your certificates expire before they are renewed, you must reinstall Linkerd and Capacity Private Cloud — after renewing the certificates — in the following order.
Uninstall Capacity Private Cloud
| helm uninstall lumenvox -n lumenvox |
Uninstall Linkerd
| export PATH=$PATH:$HOME/.linkerd2/bin linkerd viz uninstall | kubectl delete -f - linkerd jaeger uninstall | kubectl delete -f - linkerd uninstall | kubectl delete -f - |
Install Linkerd
| linkerd check --pre linkerd install --crds | kubectl apply -f - linkerd install --set proxyInit.runAsRoot=true --set proxyInit.iptablesMode=nft | kubectl apply -f - linkerd check linkerd viz install | kubectl apply -f - linkerd jaeger install | kubectl apply -f - |
Install Capacity Private Cloud
| helm install lumenvox lumenvox/lumenvox -n lumenvox -f values.yaml |
Monitoring expiration
If you use a monitoring framework such as Prometheus, you can track the time remaining before the certificates expire with a metric like this:
| sum(kubelet_certificate_manager_client_ttl_seconds/86400) |
This indicates the number of days remaining before expiration.
Renewing with a longer validity period
You can configure kubeadm to renew the Kubernetes certificates with a longer validity period, such as three years, instead of the 365-day default.
The standard kubeadm certs renew all command renews certificates with the same validity period as the originals (365 days). To set a three-year (26280-hour) validity for the renewed certificates, specify the validity period in a kubeadm configuration file.
Create a kubeadm configuration file
Create a file (for example kubeadm-config.yaml) that specifies the desired validity period. Kubernetes expresses this value in hours, so for three years you use 3 × 365 × 24 = 26280 hours:
| cat <<EOF | sudo tee ~/containers-quick-start/kubeadm-config.yaml > /dev/null apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration # 3 years * 365 days * 24 hours = 26280h certificateValidityPeriod: 26280h EOF |
Renew the certificates
Run kubeadm certs renew all, referencing the configuration file:
| sudo kubeadm certs renew all --config ~/containers-quick-start/kubeadm-config.yaml |
Stop the service and delete the old certificates
| sudo systemctl stop kubelet sudo rm /etc/kubernetes/kubelet.conf sudo ls /var/lib/kubelet/pki sudo rm /var/lib/kubelet/pki/kubelet-client-current.pem sudo rm /var/lib/kubelet/pki/kubelet.crt sudo rm /var/lib/kubelet/pki/kubelet.key |
Restore the kubelet service
| sudo kubeadm init phase kubeconfig kubelet sudo systemctl start kubelet |
Update the client configuration
| sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config |
Verify expiration
| sudo kubeadm certs check-expiration |
Security consideration. A three-year expiration reduces how often you renew, but it is generally considered less secure than the default one-year rotation. Certificate rotation is an important security control: longer-lived certificates widen the window of exposure if a private key is ever compromised.
Best practices
To keep the cluster stable, follow these practices:
- Automate certificate monitoring. Use monitoring scripts or tools like Prometheus and Grafana to assess expiration and alert you 30–60 days ahead.
- Renew before expiry. Schedule regular maintenance windows to renew certificates proactively (for example, every nine months).
- Use long-lived CA certificates where appropriate. When bootstrapping a cluster with
kubeadm, you can extend the CA certificate's validity (the CA signs all other certificates) — for example:sudo kubeadm init --certificate-expiration=87600h - Version-control and back up certificates. Always back up
/etc/kubernetes/pki, especially the CA certificates, so you can restore if a renewal goes wrong. - Rotate kubeconfig files. Update the
kubeconfigfiles (for example~/.kube/configandadmin.conf) after renewal:sudo kubeadm init phase kubeconfig admin
Automate with managed tools (optional)
To reduce operational overhead, consider:
- cert-manager: automates TLS certificate management with dynamic renewal for custom certificates (for example, for Ingress).
- Managed Kubernetes (EKS, GKE, AKS): a managed service often handles control-plane certificates for you.
Disaster recovery
If certificates expire before being renewed, restore or renew them manually:
- SSH into a control-plane node.
- Run
kubeadm certs renewto regenerate the certificates. - Restart the necessary Kubernetes components.
To avoid getting here, build alerting and monitoring, and always test your renewal process in a staging environment first. Combining proactive monitoring, scheduled renewals, and reliable backups keeps certificate expiration from disrupting your cluster.