Upgrade NGINX Ingress Versions
The ingress controller is part of your Kubernetes configuration and is managed by you, the customer, so keeping it up to date is your responsibility — and it should be done in a controlled way to minimize any interruption to connectivity into your cluster. The ingress controller routes external traffic to components inside the cluster, including the LumenVox API, so an unplanned outage here affects everything reaching the platform from outside.
The most common reason to upgrade is a security vulnerability. For example, advisory GHSA-mgvx-rpfc-9mpv describes a critical severity vulnerability affecting ingress-nginx versions older than 1.11.5. This article walks through upgrading the NGINX ingress controller in place, and rolling back if needed, using that vulnerability as a worked example.
Note: the versions shown in this article are an example and may differ from what you are running, so adjust them to your environment. Vulnerabilities and updates appear regularly; follow current best practices for keeping software patched.
Containers Quick Start
Many customers use the Capacity Private Cloud containers-quick-start repository to configure their self-managed Kubernetes cluster. Part of that process runs lumenvox-control-install.sh, which installs the ingress controller in a step like the following (line wrapping added for readability):
| ############################################# # Step 9: Install nginx ingress controller ############################################# printf "9. Installing nginx ingress controller...\n" | $TEE -a helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx \ -n ingress-nginx --create-namespace --set controller.hostNetwork=true \ --version 4.11.3 --set controller.allowSnippetAnnotations=true 1>>$MAIN_LOG 2>>$ERR_LOG if [ $? -ne 0 ]; then printf "\t\tFailed to install nginx ingress controller\n" | $TEE -a exit 1 fi |
The version installed here is 4.11.3, which is affected by the vulnerability and therefore needs to be updated.
Updating the ingress controller
You do not need to tear down and rebuild your Kubernetes cluster to upgrade the NGINX ingress controller. For this example, the goal is to move from the installed version (4.11.3, which contains NGINX 1.11.3, per the ingress-nginx documentation) to the version containing the patch (4.11.5, which contains NGINX 1.11.5), using kubectl and helm on your cluster.
1. Perform a Helm upgrade
This applies the version change described above:
| # upgrade ingress-nginx to 1.11.5/4.11.5 helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx -n ingress-nginx --create-namespace --set controller.hostNetwork=true --version 4.11.5 --set controller.allowSnippetAnnotations=true |
The new version (4.11.5) is specified toward the end of the command. Running it upgrades the Helm release, downloads and prepares any new images, and updates any ConfigMaps as needed.
Note: it may take a few seconds after this step for the new pod to appear in step 2.
Bear in mind that the parameters shown can vary between versions. The option below, for example, applies only to NGINX versions prior to 1.12, so check the latest published scripts in the containers-quick-start repository — and do your own verification — before reusing it:
| --set controller.allowSnippetAnnotations=true |
2. List the ingress pods
You need the names of the running ingress pods so you can identify which one to delete in the next step:
| # list the ingress-nginx pods kubectl get po -n ingress-nginx |
Once the upgrade from step 1 completes, listing the ingress pods looks something like this:
Here the new pod is still in a Pending (non-started) state, while the old pod keeps running until it is replaced in step 3.
3. Delete the old pod
From the pod list, identify the unique name of the old (running) controller pod and delete it so the new version can take its place:
| # the new pod won't start until we delete the old pod, so delete the old pod kubectl delete po <old-pod-name> -n ingress-nginx |
Kubernetes then replaces the old ingress pod with one running the upgraded version, which should start quickly. Once the new pod is running, verify that connectivity works as expected. Running diagnostics is a convenient check, but be sure to confirm connectivity from outside the cluster as well.
Rolling back
If you decide you would rather return to the previous version — accepting any vulnerabilities it carries — you can roll back to reverse the steps above. First, list the recent Helm history so you can see which revision to roll back to:
| helm history ingress-nginx -n ingress-nginx |
This lists the Helm revisions for the release, which may look like this:
To roll back to one of the earlier revisions shown, use:
| # The release number used in the rollback command is the revision # that was superseded by the failed deployment. helm rollback ingress-nginx <revision> -n ingress-nginx |
Note that after a rollback you again need to wait for the new pod to prepare and then delete the old ingress pod (as in steps 2 and 3 above) before the previous version is fully back in place.