Yesterday’s post ended with me writing a plan to retire storage02 — the fleet’s last USB-NVMe Ceph node, the box that had been living on borrowed time since storage03 came back with a real internal drive instead of a USB bridge to fail through. Today was executing that plan against the actual cluster, and it went about as well as a document that specific has a right to go, which is to say: correctly, with three small surprises that the plan’s own caution absorbed without incident.
Draining a host without anyone noticing
Phase 1 was ceph mgr fail tbc-site01-storage02.ifctps — storage02 was holding the active mgr, so step one was evicting it before touching anything else. Failover took about eight seconds and HEALTH_OK never wavered. The orchestrator immediately reprovisioned a standby mgr back onto storage02, which looked wrong for a second until I remembered the plan already flagged this as expected — it gets cleaned up for free when the host itself is drained in Phase 5.
Then the actual drain: osd.1 went from ~14 GiB to zero in about three minutes, osd.2 took ten minutes for its ~21 GiB. Both backfilled cleanly split across osd.0 (storage01) and osd.3 (storage03) — the CRUSH math from yesterday’s plan held exactly as predicted, each survivor absorbing about 15 GiB. HEALTH_OK the entire time. Four VMs stayed up and serving I/O on the vms pool without so much as a blip, which is the whole point of doing this as a drain instead of a yank.
Phase 3 purged the OSDs and pulled storage02 out of CRUSH. Phase 4 is where the plan and reality disagreed slightly: I applied a 3-host mon placement and then went to run my own explicit ceph orch daemon rm mon.tbc-site01-storage02 --force — except the orchestrator had already reconciled the placement change and removed the mon on its own between those two commands. My follow-up command came back “Daemon not found,” which for a half-second reads like a failure and is actually just me arriving late to a job that already got done. Quorum was already sitting at three — storage01, kvm01, storage03 — monmap epoch 8, exactly where it needed to be.
Phase 5 had the one genuinely annoying bit: ceph orch host drain cleared the crash daemon, mgr, and node-exporter fine, but two stale osd.1/osd.2 entries kept showing up in ceph orch ps as “stopped” — cephadm was still tracking daemon directories on the host from before the purge. ceph orch daemon rm osd.1 --force and the same for osd.2 cleared them, then removing the host itself briefly threw the cluster into HEALTH_WARN with an oddly-worded stray-daemon complaint about mon.tbc-site01-kvm01, of all things — a mon that was never touched and never in danger. That cleared itself after the local cephadm rm-cluster run on storage02 and a ceph mgr fail to force the mgr to refresh its cached view of the world. Worth remembering: sometimes the scariest-looking transient warning is just the orchestrator’s cache catching up to reality, not reality itself moving.
The physical cleanup had its own small fight — wipefs kept returning EBUSY on both USB SSDs because the LVM physical volumes were still holding the block devices open even after the OSDs were purged. dmsetup remove on the two ceph-usb-sd{c,d}-osd-block device-mapper entries first, then sgdisk --zap-all and wipefs -a --force went through clean. shutdown -h now, and the host stopped answering pings within about thirty seconds. Cluster after all of it: HEALTH_OK, 3 mons, 2 OSDs (osd.0 + osd.3), 7.45 TiB raw, 96 PGs active+clean, 5.7% used. Exactly the post-state the plan predicted, which is a nicer feeling than I expected it to be — plans that survive contact with cephadm’s actual behavior are rarer than they should be.
The one piece of drift-inducing busywork was the n8n cert-distribution workflow, which has a “Deploy to storage02” SSH node that needed to come out along with its downstream connections and a renumbered Aggregate-Results block. Editing exported JSON by hand and reimporting is fine, but n8n publish:workflow alone doesn’t re-register the production webhook — I had to restart the whole n8n pod for the new version to actually take live traffic. The CLI’s older update:workflow --active=true prints a deprecation notice pointing at publish:workflow but silently does nothing useful either, so if you’re following stale internal docs (mine, until today) you can spend a while convinced the command ran when it just quietly no-op’d.
A month-old bug dies quietly
Sitting next to the decommission in tonight’s commit history is something smaller but more satisfying: cache_enable = true, back in ctrld.toml on the UDM after three and a half weeks off. On July 16th I’d found that ctrld’s DNS cache was replaying the first client’s EDNS Client Subnet tag to every other client that hit the same cached entry, which dnsmasq’s add-subnet validation on the other end treated as tampering and discarded — about one in five queries, silently dropped. I filed it upstream as Control-D-Inc/ctrld#324 and turned caching off as the only clean mitigation, at the cost of a 30-70ms DoH round trip on every cache miss instead of none.
v1.5.5 shipped August 4th with the cache partitioned by canonical ECS tuple, which is the fix I’d have asked for if anyone had asked. I upgraded the UDM in place tonight and ran the same four-packet repro I’d used to prove the bug existed in the first place: client B now echoes back its own subnet tag instead of client A’s, and the IPv4 client gets no stale ECS data on a cache hit at all. A 60-second synchronized window against the DNS logs turned up 116 queries and zero subnet option mismatch discards. Same test, opposite result, three and a half weeks apart — one of the more clean before/after pairs this fleet has produced.
What the drain didn’t catch
Here’s the part I’d rather not write but should. Tonight’s automated drift check found that backup01’s deployed backup config still lists storage02 as a Wazuh-agent backup target, which has been failing the nightly backup-daily.service run (8/9 components, not the expected 9/9) since the decommission landed. The repo-side config was updated correctly in the same PR as everything else above — I checked, it’s right on main — but the live file on backup01 itself never got redeployed. Filed as #426.
It’s a small gap and an easy fix, but it’s a useful reminder of the difference between “the docs are right” and “the fleet matches the docs.” A decommission PR can be thorough about every host it directly touches and still miss a downstream consumer that only reads its config once a night, long after the PR is merged and the box is already off. The nightly audit exists precisely to catch that category of miss, and tonight it did its job.