Debugging

Tally’s kid-undo feature worked flawlessly in dev and was completely dead in production — and the reason was that the browser was faithfully running code I’d already replaced.

Two unrelated-looking outages on two different hosts — telemetry that wouldn’t ship and DNS that kept truncating — and after a full day of fixing real, secondary problems, the actual root cause of both turned out to be the same overlay network knitting the lab together.

A 500 that only one person could trigger, only on a native form submit, hidden everywhere else by client-side navigation — traced down to a compiler that emitted a call to a function it never wrote.

Layer 1 of the patch manager is officially deployed, which means today is the day I finally noticed that the healthcheck I’d been trusting for two days had been lying — politely, with a 200 OK and a copy of the React app — every time it ran.

I told myself today’s first job was the Copy Fail kernel ticket. Today’s first job turned out to be a six-hour fight with n8n’s expression parser, two failed hypotheses that landed in the repo anyway, and a deploy node that’s now structurally complete and deliberately turned off.

Two days after blaming DNS for the hourly Netbird flap and declaring it fixed, dmesg produced evidence that the real culprit was dnf-makecache.timer running on a 2GB VM with no swap.

A single transposed digit in a DNS IP address was resetting the entire Netbird mesh every 90 minutes. Closing OHP#58.

The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.

site02-kvm01 is now reachable through Netbird — not as a direct peer, but via kvm01’s subnet route. Getting there required a power cycle, a missing authorized_keys file, and rebuilding a Wazuh per-agent database from scratch.