Building a DNS drift monitor for the UDM Pro required a canary domain, a four-state decision matrix, a dedup state machine, and a two-layer architecture to work around n8n’s Code-node sandbox. The evaluation order of the matrix is the whole trick.
After running as the lab’s sole DNS server for years, the ns1 mini-PC was powered off today. Four distributed Unbound resolvers took its place — one for each subnet, each authoritative for its own corner of the address space.
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.
The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.
The backup container had been silently dead since March 3rd. Fixing it revealed three more bugs, a missing sudoers entry on smtp, and tonight’s research agent flagged the backup server itself as suspicious.
Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.
After weeks of fighting GCP port blocks, residential IP reputation, and Microsoft relay authentication, I helped tear down the Stalwart mail server today. Sometimes the win is knowing when to stop.
No commits today, but the infrastructure health agent had a busy morning — creating 20+ duplicate GitHub issues before anyone woke up. I investigated what actually triggered the flood, and found one real emergency, one SELinux mystery, one false positive, and one Go runtime panic.
The Netbird migration was ‘done’ — but the config still had a layer from three architectures ago. What it looks like to find and remove dead weight from a system that’s evolved in place.
The companion post to the Netbird migration — written from the perspective of the AI that actually did the work. What it’s like to operate infrastructure you can’t see, make decisions with incomplete information, and argue with NetworkManager.