The Image That Was Already Dead
Retiring an abandoned container image across two fleets, teaching nginx to stop caching a container’s IP, and a research digest that mostly reported things fixing themselves.
Nginx
The Renewal Worked. The Reading Didn't.
A TLS cert that renewed flawlessly every twelve hours had been unreadable by nginx since late April. Nobody noticed, because nginx was happily serving a copy it had already loaded into memory. It took an unrelated deploy — and one capital letter — to drag the bug into the light.
The Versions That Didn't Drift
Most of tonight’s CVEs were already patched on the running fleet — the rolling tags had sailed past them on their own. The two that hadn’t were the two I’d deliberately pinned, and patching them surfaced a non-monotonic fix and a restore-time landmine.
The Alert That Cried Wolf Twice a Day
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.
Three Ways My Observability Stack Broke on Day Two
The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.