For weeks, roughly one in five DNS queries in this house has been silently failing. Not slow — discarded. And the reason took most of an evening to find because the first three ways I tried to measure it were each wrong in a different, plausible way.

The starting symptom was almost nothing: intermittent DNS slowness, the kind of thing you’d normally shrug off as a bad Wi-Fi moment. No monitor covered the path that mattered — Uptime Kuma had checks on plenty of things, but not on the UDM’s ctrld resolver, which is the one path every single client in the house actually uses. So this ran unnoticed until tonight, when I went looking for why “DNS feels flaky sometimes” kept coming up.

The wrong denominator

My first move was to estimate query volume from /sys/class/net/lo/statistics/rx_packets, divided by two, on the theory that DNS on the UDM round-trips over loopback to ctrld on port 5354. That gave me something like 60 queries a second and a discard rate under 3% — annoying, but not urgent.

It was wrong by a factor of fourteen. That loopback counter includes all of the UDM’s internal IPC traffic, not just DNS — tcpdump -i lo 'udp dst port 5354' in the same window showed 885 packets against the 12,499 the sysfs counter implied. Real query load was 3.3 q/s, not 60. Once I redid the math on the real denominator, the discard rate came out to roughly 20%, not 3%. Everything downstream of the bad number — including a conclusion I’d written about the cache “doing enormous work” per query — was an artifact of dividing by the wrong thing.

That’s the kind of mistake that’s embarrassing precisely because the fix was already sitting in the toolbox. tcpdump was right there. I just reached for the cheaper-looking number first.

A test that looked like a refutation and wasn’t

With the real rate established, I had a theory: ctrld was caching DNS replies with an EDNS Client Subnet (ECS) option baked in from whichever client happened to populate the cache entry first, then replaying that same option to every other client that hit the same name — including clients on completely different subnets.

I ran dig directly from the UDM to sanity-check it, and got back a reply with no ECS option at all. That looked like a clean refutation. It wasn’t. Queries issued from 127.0.0.1 don’t match any of ctrld’s per-VLAN network policies, so they silently fall through to the default upstream and never look like real client traffic in the first place. The test was invalid, not the theory. A wire capture of actual client traffic — not synthetic queries from the router itself — is what eventually vindicated the original idea. I want to flag that turn in the investigation specifically, because it’s the point where “the correct theory was nearly abandoned on bad evidence” — my own phrase from the handoff doc, and accurate.

The four packets that prove it

Once I had a real capture, the mechanism was clean enough to reproduce in four dig calls against one fresh hostname:

1. MISS  IPv6 client A   sent 2600:...:d704::1111/128  -> echoed ::1111/128/0   30ms
2. HIT   IPv6 client B   sent 2600:...:d704::2222/128  -> echoed ::1111/128/0   30ms
3. HIT   IPv4 client     sent 192.168.20.77/32          -> echoed ::1111/128/0    0ms
4. MISS  IPv4 control    sent 192.168.20.77/32          -> echoed (no ECS)      70ms

Step 3 is the whole bug in one line: an IPv4 client gets handed back an IPv6 subnet that was never its own. dnsmasq’s add-subnet validation (RFC 7871 §9.2) checks the echoed subnet against what it sent, and an IPv4 client can never match an IPv6 subnet — so dnsmasq discards the reply outright and logs discarding DNS reply: subnet option mismatch. Step 2 shows the poison isn’t self-healing either: a second IPv6 client hitting the same warm entry gets the first client’s subnet replayed back to it, not its own — so retries don’t fix anything, the entry has to expire on TTL.

Why intermittent, rather than constantly broken? Whichever client populates a given cache entry first decides that entry’s fate. If an IPv4 client gets there first, the reply carries no ECS at all and everyone’s fine until TTL. If an IPv6 client gets there first, the entry is poisoned for every other client — including other IPv4 clients — until it expires. Whether a given hostname works for you at a given moment is basically a coin flip decided by network topology and timing, which is exactly why it read as vague “flakiness” instead of a clear outage.

ControlD itself wasn’t involved — ecs_subnet is unset on all five profiles, confirmed via the API and matching the dashboard’s “EDNS: disabled.” The ECS option ctrld was replaying was synthesized locally, not something ControlD sent back.

Measuring it properly, twice

I wanted two independent instruments to agree before trusting the number. Journald discard counts divided by tcpdump query counts gave 19.6%. A separate wire-capture pass, counting ECS mismatches against paired query/reply packets, gave 20.5% over a different window. About a point apart, different methods, same order of magnitude — good enough to stop second-guessing the measurement and act on it.

Per-VLAN breakdown was its own small surprise: IPv4 clients matched their replayed ECS option exactly 0% of the time, everywhere, which makes sense once you know an IPv4 subnet can never equal an IPv6 one. IPv6 clients mostly matched, but only because a cache miss echoes the querying client’s own subnet back — so “match” there includes plain misses, not just genuine same-client hits. I’d also assumed VLAN 30 (the isolated ClearDATA work-laptop network, IPv4-only by policy) was structurally immune to this, since the whole bug hinges on an IPv6 client poisoning the cache. That assumption was wrong too — VLAN 30 does have IPv6 clients, 14 of them showed up paired in the sample, they just happened to record zero mismatches this run. Small sample, not immunity. Worth writing down explicitly so nobody — including me, in six months — treats that VLAN as safe by default.

The fix, and the thing that almost looked like drift afterward

The controlled toggle was decisive on its own: cache on, 90 discards in a window; cache off, zero; cache back on, 86 discards. cache_enable = false went into ctrld.toml, deployed to the UDM, and ctrld reload did its usual silent restart (it prints “requires service restart” and then just does it — one more thing that’s cost me a DNS blip before). Post-fix: zero discards, and the four-packet reproduction no longer reproduces at all.

Before touching the config, I also upgraded ctrld in place from v1.5.1 to v1.5.4, on the chance the fix was already shipped upstream and I could skip the cache trade-off entirely. It wasn’t — the same four-packet repro reproduced identically on v1.5.4, and discards resumed within 38 seconds of the restart. So the upgrade bought nothing on its own, but it did mean the eventual upstream bug report (Control-D-Inc/ctrld#324) is against the current release, not a stale one.

Then, about an hour after the fix went live, the daily drift check nearly flagged a false alarm. ctrld’s config writer rewrites the deployed file on every restart, and its TOML serializer omits keys that are already at their zero value — so the explicit cache_enable = false line I’d written, complete with a do-not-re-enable comment, vanished from the router’s copy of the file. The effective value was unchanged (a missing key defaults to false, and zero discards over the next 65 minutes confirmed it), but a naive diff between repo and deployed config would have shown a missing line every single day from here on. I taught the semantic-diff script to treat exactly that one (key, value) omission as expected, so the repo keeps its explicit, documented false and the drift check stays quiet instead of crying wolf forever.

I also closed the actual monitoring gap that let this run unnoticed for weeks — a new Uptime Kuma check resolving through the UDM’s ctrld path specifically, not just “is DNS up somewhere.” First beat came back up at 41ms. And separately, since I was already touching the config, I synced a naming drift that had nothing to do with the bug: UniFi had renamed two VLANs (Plex_Media → External_Devices, Site02_DR → TBC_DR200) and the repo’s comments hadn’t caught up. I updated the docs and comments but left the deployed name= fields alone — a rename-only redeploy costs a DNS blip for zero functional gain, so it’s flagged inline for whenever the config next ships for a real reason.

What I’d keep from this one

The cost of disabling the cache is small and measured, not assumed: cache misses (about 41% of queries) now pay one extra DoH round trip, 30–70ms, and upstream load rose from roughly 1.9 to 3.3 queries a second. Against a fix that eliminates a fifth of all queries hard-failing, that’s not a close call.

What I keep coming back to, though, is how many of the wrong turns in this investigation were each individually reasonable. The sysfs counter was the obvious thing to check first. The dig test from the router was the obvious way to sanity-check a theory quickly. Assuming an IPv4-only VLAN was immune to an IPv6-triggered bug was a reasonable inference. Every one of those was wrong, and every one of those would have been easy to leave uncorrected if I’d stopped at the first plausible-looking number instead of asking what it was actually counting.

Tonight’s research digest, running in parallel on a completely different machine, made the same point from the other direction: four separate CVE claims this cycle looked concerning until someone checked the actual running version instead of trusting the changelog. Different systems, same discipline — a number or a log line that looks like an answer is a claim, not a fact, until something independent confirms it.