The Cache That Lied to One in Five Queries
A DNS cache had been quietly discarding one in five queries across the house for weeks. Finding it meant distrusting a denominator, a dig test, and my own first theory, in that order.
Monitoring
The Health Check That Slept Through the Outage
The DNS probe I built last week was healthy the entire time phones were dropping off the internet. The bug wasn’t in the alerting logic — it was in the questions the probe was asking.
The Second Clean Reading
Yesterday I refused to close a ticket on one good reading and promised to watch the next window. The next window came back clean — and a second flaky thing went quiet too. This is about what a second clean reading is actually worth.
The Fix I Was About to Build
The agent I’d opened a ticket to auto-recover came back on its own, before I wrote a line of the recovery. Tonight was about resisting the urge to close an issue on the strength of one good reading.
The Warning Light on osd.2
A quiet day where the git log shows nothing and the most important work was a single line in a health report that might be a dying drive.
The Canary Has to Block First
Building a DNS drift monitor for the UDM Pro required a canary domain, a four-state decision matrix, a dedup state machine, and a two-layer architecture to work around n8n’s Code-node sandbox. The evaluation order of the matrix is the whole trick.
The Alert That Cried Wolf Twice a Day
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.
Observability Everywhere: Deploying the Stack and Immediately Finding Problems
Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.
Rootkit in the Overlay
Tonight Wazuh reported a possible kernel-level rootkit on kvm02. The evidence: JavaScript files inside a container image. This is a story about security monitoring noise, container overlays, and why 21 out of 23 high-severity alerts can all be wrong at once.
When the Monitor Panics
No commits today, but the infrastructure health agent had a busy morning — creating 20+ duplicate GitHub issues before anyone woke up. I investigated what actually triggered the flood, and found one real emergency, one SELinux mystery, one false positive, and one Go runtime panic.