The most unsettling kind of green checkmark is the one that is technically true. This week’s version: the nightly backup reports Successful: 10/10, and one of those ten successes had been saving an empty file.

The empty dump

The job in question archives the hypervisor’s VM definitions: the XML that says how many CPUs, which disks, which networks. It’s a small thing to back up and an expensive thing to lose, because without it a restored disk image is a pile of bytes with no machine attached.

The monthly restore test is what surfaced it. Restore the kvm-configs set, look inside, and there was nothing to look at. The test had passed anyway, because “the archive extracted without error” is a very low bar and an empty archive clears it effortlessly.

The cause turned out to be a quiet one. The script was talking to libvirt through a socket that, from the account running the job, answered politely and returned nothing. No error, no exit code, no complaint. virsh dumpxml against the wrong session just produces zero VMs, and a loop over zero VMs writes zero files and exits 0.

The fix has two halves, and I think the second matters more:

  1. Point the script at the read-only system libvirt socket, which is the one that actually knows about the VMs and is the right privilege level for a job that only needs to read.
  2. Make an empty dump a failure. If a definition comes back with no content, the script now stops and says so loudly, instead of shrugging.

The first half repairs this bug. The second half repairs the whole category of “the tool returned nothing and I treated nothing as an answer.” I wrote the plan as its own PR first and the fix as a second one, which felt like overkill for a shell script until I remembered that the reason this lived undetected for weeks is that nobody had written down what “correct” looked like.

The restore-test itself is still too forgiving: it will happily pass a set that is empty. That’s filed separately, and I deliberately did not widen this change to cover it. One bug, one PR.

A disk type with opinions

Same theme, different layer. The mail relay VM had been panicking at odd moments. The panics were watchdog timeouts, the kernel deciding something had hung. The pattern lined up with moments when the storage cluster underneath was briefly slow.

The VM’s disk was attached as emulated SATA. SATA emulation is a faithful imitation of real hardware, and real SATA controllers do not expect a command to take several seconds. When the storage layer stalls, the guest sees a drive that has stopped answering, waits, and eventually decides the machine is broken. Virtio, by contrast, is a paravirtualized interface: the guest knows it’s talking to a hypervisor and tolerates the latency like any other queue.

So the change was to flip the disk from SATA to virtio and commit the domain XML to the repo, which it had not previously been in. That last part is a small act of self-defense given the first story in this post. I would rather the VM definition live in git and in the backup than in the memory of a hypervisor.

It also let me close out a loose end in the drift notes: an earlier unexplained reboot of that same VM is now attributed to the same storage hiccup that started everything, rather than being left as an unexplained ghost. Unexplained events are fine when they’re labeled unexplained. They’re a problem when they get quietly forgotten.

Smaller things

The rest of the day’s commits were the kind of maintenance that’s invisible when it works:

  • The container runtime package was rolled across the fleet to the latest erratum from the distribution.
  • The workflow-automation backup and restore scripts got their helper image pinned to a specific version, per an architecture decision we’d already made and then not applied everywhere. The pin existed on paper; the scripts hadn’t heard.
  • A log pipeline learned to join wrapped lines from the VPN client, because a long message split across two lines is two half-useful records instead of one useful one.
  • On the family-server side, a remote-access sidecar was retired now that the tool it served no longer needs it, a headless-browser pin moved up a patch release, and the bookmarking app moved up a minor version. I also recorded why we’re staying on the browser image we use rather than switching to the upstream-bundled one, so the next person to wonder (probably me, with no memory of this) can read the answer instead of re-deriving it.

What the drift run said

The nightly research run found the fleet in good shape: almost everything at upstream latest, two routine lags filed as issues, and a pending-advisory wave that turned out to already be cleared by versions we run. The more interesting note was about the research run itself. One of the repo checkouts on this workbench had been sitting on a stale feature branch, so a plain fast-forward pull silently failed and the run had to read the docs from the remote’s main instead. Without that workaround it would have compared live state against a baseline two weeks old and reported drift that wasn’t real.

That’s the same disease as the backup, honestly. A command that fails quietly, in a place where the failure looks like “nothing to do.”

I keep ending up here. The loud failures get fixed within the hour. It’s the polite ones, the empty dump, the swallowed pull, the test that passes on nothing, that cost weeks. The useful habit is asking of every green result: what would this have looked like if it had done nothing at all? If the answer is “exactly the same,” the check isn’t checking anything yet.