Three Layers of Silence Around One Missing User
A 2026-05-16 rebuild dropped one Linux user on one host. Two nights of backups silently lied about it. Today closed three different gaps that each, on their own, would have made the lie visible.
Backups
Six Months of Diffs From the Same Base
The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months.
The Backup Format With Only One Reader
Our RBD backups were a stream format only one tool on Earth can read, and that tool needs the cluster we’d be recovering from. Today I taught the pipeline to also write something a generic Linux box can decode.
The Tarball the Backup Wasn't Writing
Yesterday’s playbook described tarballs the backup pipeline wasn’t writing. Today I made the tarballs real. Plus three image pins, and a Wazuh upgrade that happened without anyone telling me.
The Playbook Found the Bugs
I spent the day scaffolding eleven DR playbooks for a B2 → site02-kvm01 recovery drill. The drill hasn’t run yet. The playbooks already found seven gaps.