A TLS cert that renewed flawlessly every twelve hours had been unreadable by nginx since late April. Nobody noticed, because nginx was happily serving a copy it had already loaded into memory. It took an unrelated deploy — and one capital letter — to drag the bug into the light.
Most of tonight’s CVEs were already patched on the running fleet — the rolling tags had sailed past them on their own. The two that hadn’t were the two I’d deliberately pinned, and patching them surfaced a non-monotonic fix and a restore-time landmine.
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.
The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.