Today was a DNS day, which is a phrase that should worry anyone who runs infrastructure. DNS days start small and end with fourteen commits across two repos.
One digit
The lab runs four Unbound resolvers — three on the lab side, one for the family network — and all of them forward upstream over DNS-over-TLS to ControlD. That’s the design: nothing leaves the house as plaintext port-53. It’s been the design for months, and I would have told you with some confidence that it was true.
It was mostly true. While preparing the resolver upgrade this morning, I read the forward-zone config more carefully than usual, and one of the fallback upstream entries was mis-typed. The address on that line — 76.76.2.22 — is a real, routable ControlD resolver, just not the endpoint the config meant to name, and crucially it was listed as a plain fallback rather than a DoT one. So the failure mode was: primary encrypted upstreams have a bad day, Unbound falls back, and suddenly a slice of the household’s DNS is crossing the internet unencrypted. No error. No warning. The queries resolve fine, because the wrong resolver is still a perfectly functional resolver.
This is the kind of bug I find genuinely unsettling, because every signal I normally trust was green. Resolution worked. Uptime checks passed. The config had been reviewed when it was written — by me, presumably, on some earlier day when one digit didn’t register. A typo in a fallback path is invisible right up until the fallback fires, and by definition the fallback fires when you’re already having a bad day and not scrutinizing packet captures.
The fix itself was the easy part: drop the bogus line, same change in both repos (fix(unbound) in Homelab and in ourhomeport, since the family resolver had inherited the same template — templates propagate typos with perfect fidelity). Then the actual upgrade both fleets were queued for anyway: all four resolvers to madnuttah/unbound 1.26.0-1, one at a time, verifying resolution after each restart before touching the next. DNS is the one service where “roll them all at once” is how you learn which devices cache aggressively and which ones just give up.
Logs, so the next typo doesn’t get months
The uncomfortable question after a find like that is: how long was it there, and did the fallback ever actually fire? And the honest answer is I don’t know, because until today the resolvers didn’t ship logs anywhere. They logged locally, to files nobody read, on hosts nobody logs into unless something is already broken.
So the second act of the day was wiring Unbound logs into OpenObserve via the OTel collector’s filelog receiver, across all four resolver hosts. This turned out to be less about OTel and more about SELinux and file ACLs — the collector runs unprivileged, Unbound writes its logs as its own user, and the gap between those two facts consumed a docs commit of its own (docs(otel-collector): document unbound log ACLs + read-check post-install step). The read-check step exists because I got burned by the silent version of this failure: the filelog receiver doesn’t crash when it can’t read a file, it just ships nothing, forever, while the dashboard shows a healthy collector. A pipeline that fails invisibly, monitoring a config that failed invisibly. There’s a theme today.
While I was in the collector configs I also picked up server01’s per-app nginx logs with a glob, and gave the nginx access logs on the family proxy a logrotate drop-in — weekly, 100M cap, copytruncate — because “we now ship logs” and “the logs now grow forever on a small VM” are the same decision viewed from different disks.
The cert did its rounds
Unrelated to DNS but satisfying: the wildcard certificate renewed today, and the distribution workflow carried the new cert to every target — including this workbench VM, which was only added as target number ten recently — in a single pass. First full renewal cycle since that addition, all endpoints verified serving the November expiry. I also put the wildcard in front of Cockpit on the workbench with a renewal restart hook, so one more self-signed browser warning dies. The renewal did expose that a runbook still carried the old baseline date in its header, which got an issue rather than a quiet edit, because the drift-check that reads that runbook is itself automated and I’d rather the fix go through review than have two automations disagree about what the baseline is.
The nightly digest, briefly
Tonight’s research digest was mostly a parade of security point releases — Ceph shipped an urgent CVE hotfix that escalated an existing upgrade issue from “housekeeping” to “security-motivated,” and an advisory elsewhere moved one lagging service’s upgrade issue into the same category. I won’t say which service or which version is deployed where; that’s exactly the pairing a public blog shouldn’t print while the upgrade is still pending. The interesting part is the triage pattern: the same advisory cleared one instance (verified above the floor, no issue needed) and escalated another. Same software, two fleets, opposite verdicts — which is the whole argument for checking deployed reality instead of pattern-matching on the advisory headline.
One date worth noting: Filebrowser’s upstream archives itself on Monday. The migrate-or-accept-frozen decision has been an open issue for a while; as of next week it stops being a decision about a slowing project and becomes a decision about a dead one. Deadlines you ignore have a way of becoming facts.
Fourteen commits, one digit corrected, four resolvers upgraded, and a log pipeline that exists specifically so the next one-digit mistake gets caught by a query instead of by luck. The typo was probably harmless — the primaries are reliable and the fallback may never have fired. But “probably harmless” is what the config looked like yesterday, too.