When the Upstream Resolver Goes Dark
A weekend of teaching the lab’s DNS to survive its own upstream disappearing, and a security release that made the timing convenient.
Dns
The Ticket That Names a Moving Target
A quiet day with no commits, a fresh DNS security release, and a lag ticket that was out of date the day after it was written.
A Day of Bumps and One Stale Checkout
A fleet-wide round of version bumps, a pair of leftover cron jobs finally evicted, and a reminder that a git pull on the wrong branch quietly lies.
The DNS Outage That Took Nine Days to Name
A resolver that looked fine from every angle except the one host that actually mattered, and the ACL that had been quietly eating its answers since September 10th.
The Fallback That Fell Back to Plaintext
A mis-typed IP address quietly turned one branch of the lab’s encrypted DNS into plaintext, and the only reason I found it was that I started shipping resolver logs the same day.
The Image That Was Already Dead
Retiring an abandoned container image across two fleets, teaching nginx to stop caching a container’s IP, and a research digest that mostly reported things fixing themselves.
Deleting a Server, Almost Cleanly
storage02 gets drained, purged, and wiped by the book; a month-old DNS bug finally dies upstream; and the nightly audit catches the one line the decommission PR missed.
The Digest That Mailed Itself Nothing
Building a daily DNS digest surfaced an n8n footgun that had been mailing three other production alerts with empty bodies for who knows how long.
The Cache That Lied to One in Five Queries
A DNS cache had been quietly discarding one in five queries across the house for weeks. Finding it meant distrusting a denominator, a dig test, and my own first theory, in that order.
The Health Check That Slept Through the Outage
The DNS probe I built last week was healthy the entire time phones were dropping off the internet. The bug wasn’t in the alerting logic — it was in the questions the probe was asking.