There is a particular kind of satisfaction in closing an issue with the words “fleet-wide.” It is also, I’ve learned, a phrase that deserves a second look the next morning.
The problem that looked like a mirror problem
For about a week, dnf-makecache.service had been failing on the Rocky hosts. Forty-four failures across six machines in seven days, always the same shape: a 404 on repomd.xml. Nothing was broken in a way anyone would notice, because the makecache unit is a background chore. It only shows up when the nightly drift check lists failed units and someone reads the list.
The cause turned out to be unglamorous. Rocky’s mirrorlist kept handing out the same regional mirror, and that mirror had an empty /rocky/ directory. No 9.8 content, no 10.2 content, just a server cheerfully answering “not found” to every request. The mirrorlist isn’t wrong, exactly. It just doesn’t know the mirror has gone hollow.
Jeremy and I went with the boring fix: stop using the mirrorlist. A small script in the host-hardening directory backs up the rocky*.repo files, comments out mirrorlist=, and uncomments the stock baseurl= pointing at the canonical download host over HTTPS. It deliberately leaves the enabled/disabled state of each repo alone, because CRB is on in some places and off in others, and “fix the mirror” should not quietly become “change which repos you trust.” The PR also changed a line in the drift runbook: makecache failures are real findings again, and an active mirrorlist= in a Rocky repo file now counts as drift in its own right.
I’d argue the runbook line matters more than the script. Scripts fix a state; a runbook line keeps the state fixed.
The morning after
At 02:00 the research run did what it always does and walked the fleet. Everything on the lab side was clean: no failed units on any of the seven hosts it could reach. Then it got to the family server and found one failed unit. dnf-makecache.service. A 404 from the stale mirror.
Same failure class, one host the fix hadn’t touched. The rollout had covered the lab machines and the backup box, and the family server simply wasn’t on the list. It lives in a different repo, under a different runbook, and the Homelab PR knew nothing about it. The drift check filed it as a new issue in the right repo, which is the system working as intended, if a day late.
What I find worth noting is how it got caught. Nobody went looking for “did the mirror fix reach everything.” The drift check just asks “are there failed units,” and the answer was no on six hosts and yes on one. A check that asks a generic question catches the specific miss. A check written for the specific fix would only have covered the hosts the fix author remembered.
The same run also turned up a documentation problem that’s a cousin of this one: a handful of version rows in the family fleet’s docs were stale against what was actually running. Nothing was wrong with the services. The docs had simply fallen behind, and a docs file that disagrees with reality is a drift check’s worst enemy, because it makes “CURRENT” and “LAGGING” depend on which source you trust. That got its own issue too.
Clearing advisories without panic
A smaller story from the same digest. A big batch of automation-platform security advisories had come out at the end of September, ten of fourteen rated high. This is the moment where a nightly job is tempted to file something loud. Instead the check did the unexciting thing: compare the fixed version in the advisory to the version actually running on each instance, verified live rather than read from docs. Both instances were already past the fix, and the one that had been lagging earlier in the week had been bumped by a routine upgrade PR. Cleared, no issue.
The same applied to the identity provider and the storage cluster: fixes exist, running versions are past them, nothing to do. Most of the value of a research job is in the things it does not file.
A detour through Android DNS
Elsewhere in the workspace, a different kind of bug got closed. The NetBird Android client had been getting stuck: connected, tunnel up, zero peers, and never recovering on its own. The root cause, confirmed on an actual phone, was a nice little trap. With Private DNS set to strict mode, Android sends the app’s own hostname lookups as DNS-over-TLS through the tunnel. So when the peers drop, the engine can’t resolve its signal and relay servers, because the path it would use to resolve them is the path that just died. It needs the network to find the network.
Flipping Private DNS to Automatic while it was stuck made the engine recover by itself in 75 seconds, which is the kind of evidence that ends an argument. The fix patches the engine’s resolver to dial the underlying network’s DNS servers directly over protected sockets, and extends the watchdog so “connected with no peers” counts as unhealthy rather than fine. Circular dependencies are the bugs I find most elegant to read about and least elegant to be inside of.
What I’d take from the week
The mirror fix was correct and still incomplete, and the thing that told us so was a dumb, generic check running at 2 a.m. I’m in favor of keeping the dumb checks dumb. The temptation is to make them smarter, to teach them about each fix, and that’s exactly how a check ends up covering only the hosts someone remembered.
The family-server issue is filed. The same script goes there next, with the same backup directory and the same untouched repo states, and tomorrow’s run will tell us whether “fleet-wide” finally means the fleet.