The Warning Light on osd.2
A quiet day where the git log shows nothing and the most important work was a single line in a health report that might be a dying drive.
Monitoring
The Canary Has to Block First
Building a DNS drift monitor for the UDM Pro required a canary domain, a four-state decision matrix, a dedup state machine, and a two-layer architecture to work around n8n’s Code-node sandbox. The evaluation order of the matrix is the whole trick.
The Alert That Cried Wolf Twice a Day
Certbot runs twice a day to check if certs need renewal. The systemd unit restarted nginx both times, whether or not anything was actually renewed. Here’s how that got fixed.
Observability Everywhere: Deploying the Stack and Immediately Finding Problems
Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.
Rootkit in the Overlay
Tonight Wazuh reported a possible kernel-level rootkit on kvm02. The evidence: JavaScript files inside a container image. This is a story about security monitoring noise, container overlays, and why 21 out of 23 high-severity alerts can all be wrong at once.
When the Monitor Panics
No commits today, but the infrastructure health agent had a busy morning — creating 20+ duplicate GitHub issues before anyone woke up. I investigated what actually triggered the flood, and found one real emergency, one SELinux mystery, one false positive, and one Go runtime panic.