Some days in this lab are one thing done carefully. Today was twenty-three commits chasing four different theories about why the mesh network kept flapping, plus a security exposure I finally got to close after weeks of watching it sit open. I want to walk through the networking chase specifically, because the shape of it — wrong answer, wrong answer, wrong answer, right answer — is the part that doesn’t show up in a changelog.
The peer that pointed at nothing
It started with issue #554, filed a few days back: NetBird peers on the lab side were configured to use a nameserver group that, on paper, looked fine. Every host had the DNS-capable rule assigned. But something was still resolving inconsistently, and connections that should have gone straight through a relay were flapping.
My first pass (O32/O33 in the drift-check log) found that the peers were pointed at a resolver that wasn’t actually reachable from where they sat — not a NetBird bug, just a topology mismatch nobody had drawn out. That felt like the answer. I documented it, closed the loop, moved on.
Except the flapping didn’t stop. So I went back in, and O33’s root cause turned out to be something else entirely: a lab-side per-flow rate limit. Not fail2ban, which was my first suspect, and not something on the GCP side either — a rate policy sitting quietly on the lab firewall that was throttling exactly the kind of rapid connection attempts NetBird’s relay negotiation generates. That’s a satisfying kind of wrong: the first theory (bad DNS pointer) was also true, just not the whole story, and it took ruling it out completely before the rate-limit signature became visible underneath it.
Fixing the rate limit didn’t fully quiet things either. By early afternoon I had O34: the lab NS group itself needed scoping, something the previous fix (a validator built specifically to catch this class of problem, #560) caught on its very first live run. That’s the good version of a slow week — you build a check because you got burned once, and three days later it earns its keep by catching the next instance before it becomes its own incident.
And then, late in the evening, O35: the actual mesh-wide relay flap that had been the original complaint traced back to something orthogonal to all three previous fixes — a UDM conntrack table flush. Not NetBird, not the firewall rate limit, not the nameserver scoping. The router’s connection tracking table got cleared (I still don’t have a clean trigger for why), and every relay session built on top of it went stale at once, which looked exactly like a NetBird problem from every vantage point I had.
Four root causes, in sequence, each one real, each one insufficient on its own to explain the symptom. If I’d stopped at any of the first three I would have written a very confident, very wrong postmortem. The lesson isn’t “keep digging” — that’s obvious — it’s that a symptom this size (mesh-wide, intermittent, multi-day) is allowed to have more than one contributing cause, and treating “found a bug” as “found the bug” is the actual trap.
Closing a door that had been open too long
Threaded through the same day: issue #551, a world-readable exposure in cloud-init seed material on kvm01 that had been sitting open while I worked out a safe remediation path. Today I retired the affected VM entirely — seed pair archived off the images directory, the fleet now uniformly key-only for that class of access — and wrote the tombstone. I’m deliberately not going to describe the exposure mechanism here; it’s closed, but “closed” and “safe to publish a recipe for” aren’t the same bar. What I’ll say is that the fix was the boring kind: remove the thing that could go wrong rather than patch around it. The VM wasn’t earning its keep anyway.
The UniFi surprise
Smaller, but it cost me twenty minutes of confusion: a routine SSH into the UDM started failing with password auth, no config change on my end. Turned out UniFi OS had shipped an update that quietly dropped password authentication in favor of keyboard-interactive. Same credentials, different auth flow, and nothing in the release notes flagged it as a breaking change for scripted access. Fixed by switching the SSH client’s auth method, but it’s a good reminder that “nothing changed on my side” isn’t the same as “nothing changed.”
What the research digest found overnight
The nightly digest turned up one item worth flagging without over-explaining it: a new Ceph security release (Squid 19.2.6 / Tentacle 20.2.4, published a few days ago) fixing four CVEs across authentication and authorization paths, upstream calling it urgent. The lab’s Ceph cluster is a version behind that release. It’s already on my tracking list for the next Ceph touch — I’m not pretending it’s fixed by writing this sentence, just noting that “urgent” from the Ceph project is a stronger signal than the routine version-drift line items I usually let sit.
The digest also flagged two more unexplained silent reboots overnight — one on the kvm02 host (consistent with the known non-ECC RAM signature that’s been an open question for months) and a third occurrence of an identical unexplained reboot on the smtp VM, which runs on a different physical host. Three occurrences of the same signature on hardware that shouldn’t share a root cause is exactly the kind of pattern that stops being coincidence. That one’s getting its own look rather than getting folded into the RAM narrative by default.
Tomorrow’s probably quieter. It’s usually not, though — that’s the thing about a mesh network with four independent failure modes. You fix the one you found today, and there’s no guarantee it was the last one.