Debugging

Chasing an intermittent NetBird disconnect that looked like resource exhaustion and turned out to be a relay bug two versions back — plus the paperwork that comes after.

A wildcard cert renewal failed on three of ten targets today — and the status email said Complete anyway, because that’s the only word it knew how to say. The fix was two unrelated latent bugs, and a subject line that finally learned to admit failure.

Tally’s kid-undo feature worked flawlessly in dev and was completely dead in production — and the reason was that the browser was faithfully running code I’d already replaced.

Two unrelated-looking outages on two different hosts — telemetry that wouldn’t ship and DNS that kept truncating — and after a full day of fixing real, secondary problems, the actual root cause of both turned out to be the same overlay network knitting the lab together.

A 500 that only one person could trigger, only on a native form submit, hidden everywhere else by client-side navigation — traced down to a compiler that emitted a call to a function it never wrote.

Layer 1 of the patch manager is officially deployed, which means today is the day I finally noticed that the healthcheck I’d been trusting for two days had been lying — politely, with a 200 OK and a copy of the React app — every time it ran.

I told myself today’s first job was the Copy Fail kernel ticket. Today’s first job turned out to be a six-hour fight with n8n’s expression parser, two failed hypotheses that landed in the repo anyway, and a deploy node that’s now structurally complete and deliberately turned off.

Two days after blaming DNS for the hourly Netbird flap and declaring it fixed, dmesg produced evidence that the real culprit was dnf-makecache.timer running on a 2GB VM with no swap.

A single transposed digit in a DNS IP address was resetting the entire Netbird mesh every 90 minutes. Closing OHP#58.

The monitoring stack I deployed yesterday started lying to me within 24 hours. Here’s how I chased down three separate failures in one morning.