I want to tell you about a bug that was technically visible the entire time and still took nine days to name, because the evidence kept pointing at the wrong layer.
Some background, since the timeline matters here. On September 10th, kvm02’s memory started throwing errors under Memtest86+ — 31 errors in 33 minutes as a matched pair, clean when run one module at a time. Not a code problem, a silicon problem, and the fix was a hardware evacuation: move every application container off kvm02 and onto kvm01 while the bad RAM pair got sorted out. That included lab-dns, one of two Unbound resolvers that answer DNS for the 100.168.192.in-addr.arpa zone on VLAN 100. Its sibling, lab-dns-2, was already living on kvm01. So after the move, both resolvers sat on the same host, at .53 and .153, both bridge0 aliases on the same box.
That’s not wrong by itself. Redundant resolvers on one host during an emergency evacuation is a completely reasonable tradeoff — you’re solving for “don’t run application containers on RAM that fails Memtest,” not for resolver placement. The plan even had a Phase 4 to move lab-dns back to kvm02 once it had run clean for a week. Nobody thought DNS placement was the risk here.
But starting around September 10th, server01 — the OurHomePort application host — started reporting Nameservers: 1/2 Available in its NetBird status. One resolver down, one up. Around 5,800 upstream timeouts a day. Not a total outage — the healthy resolver picked up the slack — but a real, sustained degradation, the kind that shows up as slow page loads and the occasional failed internal lookup rather than a dramatic red alert.
Why it took so long to pin down
The obvious suspect was the Android NetBird app, version 0.6.0, which had shipped right around when the symptom started. Coincident timing is a hell of a drug. I want to be honest that this suspicion got real attention before it was cleared — checking whether the phone’s peer config had changed, whether its ACL group membership had shifted, whether some client-side DNS-over-NetBird setting had flipped in the new build. None of that panned out. The phone’s peer was unchanged and had a full ACL to both resolvers the whole time. It was innocent, and proving that took real effort that produced a null result — the least satisfying kind of debugging, but not wasted, because it closed off the wrong branch of the investigation for good.
With the phone cleared, the actual culprit was sitting in a spot nobody was watching: NetBird’s INPUT ACL rules. The ohp-servers group (server01’s group) had a policy allowing it to reach lab-servers (kvm01, kvm02, and friends) — but only for TCP 1514, 1515, and 5080. Those are Wazuh and OpenObserve ports. There was no rule opening port 53 for that path.
Before September 10th, this didn’t matter, because lab-dns lived on kvm02, and traffic to kvm02 apparently followed a different route — the lab-trusted-full subnet FORWARD path — that wasn’t gated by that narrow ACL. Once lab-dns moved to kvm01, its queries from server01 started hitting kvm01’s INPUT ACL instead, and got silently dropped, because DNS wasn’t in the allowed port list. The queries arrived — a tcpdump on kvm01’s wt0 interface confirmed that — they just never got answered. That’s a much quieter failure mode than “resolver crashed” or “network partition.” From server01’s point of view it looked exactly like a resolver that was reachable but flaky, which is exactly what “1/2 Available” reports.
The fix, and the piece that’s still open
The actual remediation was almost anticlimactic once the cause was clear: execute Phase 4 of the evacuation plan early. lab-dns moved back to kvm02 on September 13th — the exact same fix that was already scheduled, just triggered by “we found the bug” instead of “kvm02 has had a clean week.” lab-dns-2 covered the gap on kvm01 the whole time, which is presumably why nobody got paged. Verification was the boring, satisfying kind: dig @192.168.100.53 answering from three different vantage points, netbird status on server01 flipping to 2/2 Available, and confirming the systemd units and bridge aliases matched on both hosts.
What’s interesting to me is what’s not fixed. lab-dns-2 at .153 is still sitting on kvm01, still behind that same narrow INPUT ACL, still unreachable from ohp-servers on port 53. It’s covered right now because lab-dns at .53 is back on kvm02 and takes the healthy path — but if kvm01 ever needs to host both resolvers again, even briefly, the same failure comes back, and this time nobody would have a “well it used to work” baseline to compare against. That’s logged as open work: a narrow ohp-to-lab-dns policy opening UDP and TCP 53 between the two groups, so resolver placement stops being load-bearing for whether cross-VLAN DNS resolves at all. It hasn’t been built yet because the immediate symptom is gone and there’s no fire under it — which is exactly the kind of thing that turns into next September’s nine-day mystery if it sits too long.
The thing I keep turning over is that every individual step in this chain was reasonable. Evacuating bad RAM was correct. Putting both resolvers on the surviving host temporarily was correct. The ACL scoping ohp-servers down to just the ports it needs was correct security practice, not an oversight — nobody set out to block DNS, they set out to not open a bunch of unrelated ports. The bug only exists in the gap between three independently sound decisions, which is the same shape as a lot of interesting infrastructure failures: nothing is wrong, and yet something is broken.