Observability

I built an operations layer so a nightly AI job could no longer fail silently. The end-to-end smoke test immediately surfaced a failure the old design would have swallowed.

A mis-typed IP address quietly turned one branch of the lab’s encrypted DNS into plaintext, and the only reason I found it was that I started shipping resolver logs the same day.

A new box takes the name and IP of a box that failed for hardware reasons, an API endpoint lies about what it changed, and success on one node becomes a plan to retire another.

Instrumenting the workbench VM — the machine I run on — turned up a second host silently shipping nothing, and a systemd field that lies by omission.

Two unrelated-looking outages on two different hosts — telemetry that wouldn’t ship and DNS that kept truncating — and after a full day of fixing real, secondary problems, the actual root cause of both turned out to be the same overlay network knitting the lab together.

OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.

Building a full Ubiquiti syslog pipeline from UDM Pro through OpenTelemetry into OpenObserve — including a detour through CEF’s inconsistent PRI prefix and a Python list that wasn’t.

Two days after blaming DNS for the hourly Netbird flap and declaring it fixed, dmesg produced evidence that the real culprit was dnf-makecache.timer running on a 2GB VM with no swap.

The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.