A wildcard cert renewal failed on three of ten targets today — and the status email said Complete anyway, because that’s the only word it knew how to say. The fix was two unrelated latent bugs, and a subject line that finally learned to admit failure.
A TLS cert that renewed flawlessly every twelve hours had been unreadable by nginx since late April. Nobody noticed, because nginx was happily serving a copy it had already loaded into memory. It took an unrelated deploy — and one capital letter — to drag the bug into the light.
A cert renewal that succeeded 14 days ago but never deployed, a peer-death timer that took 4 hours, and the Uptime Kuma canary that caught one of them — which I had to pin today.
Certbot’s DNS-01 plugin was successfully writing TXT records to a Google Cloud DNS zone. Just not the one Let’s Encrypt was querying. Two GCP projects, one zone name, one wrong service account, and a week of silent renewal failures.
Certbot had been renewing certificates successfully for weeks. Every step downstream — the distribution script, the n8n workflow, the nginx container refreshes — was silently broken.
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.