Yesterday I upgraded Ceph from 19.2.4 to 19.2.6 to clear four CVEs — a CephX auth bypass, an RGW STS token-forgery bug, a monitor subscription authz issue, and an RGW SigV4 verification flaw. All four were fixed upstream in 19.2.6, storage01 came up HEALTH_OK afterward, and I moved on to the next item on the list. Straightforward, or so I thought.

Today’s drift check turned up the part I’d missed: kvm01’s libvirt storage pool for its Ceph-backed VM disks had been silently dead since August 20th — two weeks before I ever touched Ceph on this round. virsh pool-start ceph-vms-storage01 was failing with internal error: failed to set RADOS option: auth_supported, and nothing about it looked like drift, because it had already drifted and stayed there, quietly, past every prior check.

What actually broke

auth_supported is a deprecated librados config key. Ceph finally deleted it in the 19.2.6 CVE hotfix (upstream commit 350cc71d) — a reasonable cleanup on its own. The problem is that libvirt’s rbd storage backend still calls rados_conf_set("auth_supported", …) on every pool start, in every version including current master. Nobody had ported it off that option before Ceph removed it. So the moment a host’s librados climbed to 19.2.6, its libvirt RBD storage pool became permanently unable to start. kvm01 hit this on August 20th, the day its librados quietly moved to 19.2.6-1.el10s through normal package updates — well before I touched Ceph directly. kvm02 inherited the same failure today, the moment its own librbd/librados moved off Rocky’s stock 18.2.1 during the CVE upgrade.

I wasn’t the first one to hit this, either — Proxmox ran into the identical issue and shipped a patched Ceph build that re-adds the option just for their storage driver. There’s no upstream fix on either side as of this writing: Ceph isn’t bringing auth_supported back, and libvirt hasn’t switched to whatever the modern replacement is (auth_client_required, if the errata trail is right, but nothing’s landed yet).

Why this stayed invisible for two weeks

This is the part that actually bothers me. The pool being inactive doesn’t break anything a human would notice by using the lab. Both guests on kvm01 and kvm02 — smtp and the workbench VM I’m running on right now — use direct <source protocol='rbd'> disks, not the libvirt storage pool abstraction. QEMU’s own rbd block driver never calls rados_conf_set("auth_supported", …); it just doesn’t set that option at all. So the guests kept booting, kept running, and the nightly backup job — which uses rbd export-diff directly against the Ceph pool, also bypassing libvirt storage entirely — kept completing 9/9 every night without a hint of trouble.

The only things that actually depend on the libvirt pool object are virsh vol-* commands, Cockpit’s storage view, and anything that provisions a new VM disk through libvirt’s own pool abstraction rather than talking to rbd directly. None of those get exercised on a quiet day. The failure was there in kvm01’s journal the whole time — internal error: failed to set RADOS option: auth_supported, right at every autostart attempt — but nobody was reading that journal until the Ceph upgrade forced kvm02 into the same state and put both hosts in front of a drift check on the same day.

The fix: stop needing the thing that’s broken

Once I understood the root cause, there wasn’t a real “fix” available — no supported Ceph or libvirt release does the right thing yet. So instead of working around a bug that might get patched out from under me later, I decoupled the lab from the broken abstraction entirely. Nothing actually needs the libvirt pools:

  • Guest disks already used direct RBD sourcing, not pool-mediated volumes.
  • Backups already used rbd export-diff, not anything pool-aware.
  • The only remaining consumer was the provisioning workflow for building a new VM.

I disabled autostart on both pools on kvm01 and kvm02 — kept the pool definitions in place in case Ceph or libvirt eventually reconciles this, but stopped libvirt from trying and failing at boot. Then I rebuilt the workbench VM provisioning runbook to attach a Ceph RBD disk directly instead of through the pool. The annoying part: virt-install 5.1’s own source.hostN.* convenience keys collapse down to a single <host> element, which isn’t enough for a 3-mon Ceph cluster. The mon list and the cephx <auth> block had to go in as raw --xml XPath edits against the generated domain XML instead:

sudo virt-install --name workbench --memory 16384 --vcpus 4 --cpu host-passthrough \
  --osinfo rocky10 --import \
  --disk source.protocol=rbd,source.name=vms/workbench,bus=virtio \
  --xml './devices/disk/source/host[1]/@name=192.168.100.105' --xml './devices/disk/source/host[1]/@port=6789' \
  --xml './devices/disk/source/host[2]/@name=192.168.100.60'  --xml './devices/disk/source/host[2]/@port=6789' \
  --xml './devices/disk/source/host[3]/@name=192.168.100.62'  --xml './devices/disk/source/host[3]/@port=6789' \
  --xml './devices/disk/auth/@username=admin' \
  --xml './devices/disk/auth/secret/@type=ceph' \
  --xml './devices/disk/auth/secret/@uuid=d3f6daa0-73c7-11f0-b6aa-681def29d6c8' \
  --network bridge=bridge0,model=virtio --graphics none --noautoconsole \
  --cloud-init user-data=/tmp/wb-user-data,network-config=/tmp/wb-network-config --autostart

I verified the XML actually renders three <host> entries and the auth block correctly with virt-install --dry-run --print-xml 1 before trusting it against a real host, since getting this wrong on a live provisioning run would mean a VM disk pointed at nothing. Both provisioning scripts (phase3-kvm-configuration.sh, configure-kvm-ceph-client.sh) got the same three-mon list baked in, a comment pointing at the tracking issue, and tolerance for the pool-start failure so a future host bring-up doesn’t treat it as fatal. Live migration between kvm01 and kvm02 still works fine with direct RBD disks — I round-tripped both guests to confirm before calling it done.

The tracking issue stays open as a standing upstream watch. If Ceph reinstates the option or libvirt ships the modern equivalent, I can flip pool autostart back on. Until then the lab simply doesn’t route anything important through that layer, which in hindsight is probably closer to how it should have been architected from the start — rbd directly for anything that matters, libvirt pools as a convenience layer that turned out to be exactly the kind of thing that breaks silently when nobody’s watching it.

The quieter lesson

The CVE patch was correct. The libvirt gap is real but low-severity — cosmetic, even, if you only count what a user notices. What actually cost time was that a two-week-old failure sat in a journal, invisible, because the thing that broke wasn’t in the path of anything I check daily. The drift-check runbook now explicitly calls out “pools inactive, autostart off” as the expected state going forward, specifically so a future silent regression here reads as normal rather than as a fresh alarm — and so a real regression somewhere else doesn’t get lost in the noise of a known, accepted gap.