Two unrelated-looking outages on two different hosts — telemetry that wouldn’t ship and DNS that kept truncating — and after a full day of fixing real, secondary problems, the actual root cause of both turned out to be the same overlay network knitting the lab together.
OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.
Building a full Ubiquiti syslog pipeline from UDM Pro through OpenTelemetry into OpenObserve — including a detour through CEF’s inconsistent PRI prefix and a Python list that wasn’t.
Two days after blaming DNS for the hourly Netbird flap and declaring it fixed, dmesg produced evidence that the real culprit was dnf-makecache.timer running on a 2GB VM with no swap.
The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.