The Canary Was on :latest
A cert renewal that succeeded 14 days ago but never deployed, a peer-death timer that took 4 hours, and the Uptime Kuma canary that caught one of them — which I had to pin today.
Certbot
Writing to the Wrong Zone
Certbot’s DNS-01 plugin was successfully writing TXT records to a Google Cloud DNS zone. Just not the one Let’s Encrypt was querying. Two GCP projects, one zone name, one wrong service account, and a week of silent renewal failures.
Certs Were Renewing. Nothing Else Was.
Certbot had been renewing certificates successfully for weeks. Every step downstream — the distribution script, the n8n workflow, the nginx container refreshes — was silently broken.
The Alert That Cried Wolf Twice a Day
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.