How to renew the speech-tls-secret certificate

Every externally exposed endpoint on a Capacity Private Cloud cluster, including the APIs and both portals, presents one TLS certificate, and that certificate lives in a single Kubernetes secret named speech-tls-secret. When it expires, every browser, gRPC client, REST client, and MRCP API server fails its TLS handshake at the same moment, so an expiring certificate looks like a total platform outage even though every pod is healthy. This article explains how to replace the certificate in place, with no downtime on the cluster, and how to update the systems that hold their own copy.

Item	Detail
Applies to	Capacity Private Cloud container installs that use the Linkerd service mesh and the nginx ingress controller (Quick Start / kubeadm, GKE, and EKS)
Secret and namespace	`speech-tls-secret` in the `lumenvox` namespace
Symptom	Browsers, gRPC/REST clients, or the MRCP API server report an expired or invalid certificate for the platform hostnames
Downtime	None expected on the cluster. The ingress controller reloads the secret in place. MRCP API servers do require a restart (see Update clients that hold a copy of the certificate).

How the certificate is used

speech-tls-secret is a Kubernetes TLS secret (type kubernetes.io/tls) that holds the certificate (tls.crt) and the private key (tls.key). The nginx ingress controller uses it to terminate HTTPS for every externally exposed hostname:

lumenvox-api (gRPC)
biometric-api
management-api
reporting-api
admin-portal
deployment-portal

Each hostname is the name above followed by the hostnameSuffix configured for your installation, for example admin-portal.example.com.

The secret is created once during installation with kubectl create secret tls. Helm does not manage it, so helm upgrade never changes it and nothing renews it automatically. When the certificate expires, you must replace it manually.

Self-signed versus custom certificates. The installer can generate a self-signed certificate that is valid for 10 years (3650 days), which makes expiry easy to forget until it happens. If you supplied your own certificate, for example from an internal or public CA, its validity period is set by that CA and is typically one year or less. This article covers replacing either type with a new certificate and key. To generate a new self-signed certificate, see Setting up self-signed certificates for the portal.

Before you begin

You need:

kubectl access to the cluster, with permission to edit secrets in the lumenvox namespace. For a refresher, see How to use kubectl.
The new certificate file (PEM) containing the server certificate followed by any intermediate CA certificates (server.crt).
The unencrypted private key (PEM) that matches the new certificate (server.key). kubectl rejects passphrase-protected keys, so remove the passphrase first.
A new certificate that includes Subject Alternative Names (SANs) for every platform hostname in use (see the list above).

Note: If your CA re-issued the certificate against the same private key, the key file is the same as before. If the CA issued a new key pair, use the new key.

Procedure

Step 1: Check the current certificate

Confirm the expiry date of the certificate currently stored in the secret:

kubectl get secret speech-tls-secret -n lumenvox \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -enddate

Step 2: Validate the new certificate and key

Check the expiry date and SANs of the new certificate:

openssl x509 -in server.crt -noout -subject -enddate -ext subjectAltName

Confirm that the key matches the certificate. Both commands must print the same hash:

openssl x509 -in server.crt -noout -pubkey | sha256sum
openssl pkey -in server.key -pubout | sha256sum

Step 3: Back up the existing secret

kubectl get secret speech-tls-secret -n lumenvox -o yaml > speech-tls-secret.backup.yaml

Store the backup securely, because it contains the private key.

Step 4: Replace the secret in place

Use the dry-run and apply pattern. Running kubectl create secret tls directly fails with AlreadyExists, and deleting the secret first causes a short outage.

kubectl create secret tls speech-tls-secret \
  --key server.key --cert server.crt \
  -n lumenvox --dry-run=client -o yaml | kubectl apply -f -

Expected output: secret/speech-tls-secret configured

Step 5: Reload the ingress controller (only if needed)

The nginx ingress controller normally detects the change and reloads within a few seconds. If the old certificate is still being served after a minute or two, restart the controller:

kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx
kubectl rollout status deployment/ingress-nginx-controller -n ingress-nginx

Step 6: Verify

Check which certificate is served for each hostname, replacing the hostname with your own:

openssl s_client -connect lumenvox-api.example.com:443 \
  -servername lumenvox-api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -enddate

Repeat for admin-portal, deployment-portal, management-api, reporting-api, and biometric-api. If DNS is not configured, the client machine needs hosts file entries that point to the cluster IP (see the hosts file section of Setting up a Deployment).

Then confirm that the Admin Portal loads in a browser (use a private or incognito window to avoid cached certificates) and that an API call, such as a gRPC test from the sample scripts, succeeds.

Update clients that hold a copy of the certificate

Replacing the secret does not update systems that were given their own copy of the certificate. Update each of the following as applicable.

MRCP API server

When the MRCP API server connects to lumenvox-api over TLS, it keeps a copy of the server certificate in mrcp-api/docker/certs/. Perform these steps on every MRCP API server that connects to the cluster. For installation background, see Setup of MRCP API Server.

Plan for a maintenance window. Running docker compose down stops the MRCP service and drops any calls or sessions in progress. If you run several MRCP API servers, update them one at a time so the others continue to serve traffic.

1. Go to the certs directory

Change to the certs directory inside the mrcp-api installation (the location where the repository was cloned):

cd mrcp-api/docker/certs

2. Copy in the new certificate

Keep the existing file name (server.crt) so the existing configuration continues to work. Optionally keep a copy of the old file first:

cp server.crt server.crt.old
cp /path/to/new/server.crt server.crt

3. Restart the MRCP API containers

Run the restart from the mrcp-api/docker directory:

cd ..
docker compose down && docker compose up -d

4. Verify

Confirm that the containers are running and connecting to the speech API without certificate errors:

docker compose ps
docker compose logs --tail 50

Other clients

Component	Action
Windows or desktop trust (self-signed or private CA)	If the certificate was manually installed in Trusted Root Certification Authorities, install the new certificate or the new CA root the same way.
Custom gRPC/REST clients	Any client that pins the certificate or trusts a specific CA bundle must be updated with the new certificate or CA.

Troubleshooting

Symptom	Likely cause and fix
`failed to find any PEM data` or `private key does not match public key`	The key and certificate do not match, or the key is passphrase-protected. Re-run the hash comparison in Step 2, and remove any passphrase with `openssl rsa -in encrypted.key -out server.key`.
Browser shows "Kubernetes Ingress Controller Fake Certificate"	The ingress could not load the secret because of invalid contents or a wrong name or namespace. Confirm the secret exists with `kubectl get secret speech-tls-secret -n lumenvox`, and check the controller logs with `kubectl logs -n ingress-nginx deploy/ingress-nginx-controller`.
Old certificate still served	Wait briefly, restart the controller (Step 5), and test from a private browser window. Confirm that the secret itself holds the new certificate by using the command in Step 1.
Clients fail with "unable to verify the first certificate"	The intermediate CA certificates are missing. Rebuild `server.crt` with the server certificate first, followed by the intermediates, and re-apply it.
Hostname mismatch errors	The new certificate is missing a SAN for that hostname. Request a re-issued certificate that includes all platform hostnames.

Rollback

If the new certificate causes problems and the old one is still valid, restore the backup from Step 3:

kubectl apply -f speech-tls-secret.backup.yaml

If kubectl reports a conflict, remove the resourceVersion and uid fields from the backup file first.

Was this article helpful?