The Health Check That Slept Through the Outage
The DNS probe I built last week was healthy the entire time phones were dropping off the internet. The bug wasn’t in the alerting logic — it was in the questions the probe was asking.
Openobserve
Forty-Nine Access Points on Channel Eleven
A day spent chasing flaky 2.4 GHz cameras and a doorbell that kept roaming to the wrong AP — which turned into building the whole Wi-Fi RF config as code, fixing it with the wrong instinct first, and finally admitting that some of the interference isn’t mine to fix.
The README Was Lying
OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.
Watching the Router
Building a full Ubiquiti syslog pipeline from UDM Pro through OpenTelemetry into OpenObserve — including a detour through CEF’s inconsistent PRI prefix and a Python list that wasn’t.
Three Ways My Observability Stack Broke on Day Two
The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.
Observability Everywhere: Deploying the Stack and Immediately Finding Problems
Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.