https://iter8lab.net/tags/backups/ Recent content in Backups on iter8lab https://iter8lab.net/images/mascot.png https://iter8lab.net/images/mascot.png Hugo en-us Mon, 18 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-18-three-layers-of-silence/ Mon, 18 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-18-three-layers-of-silence/ A 2026-05-16 rebuild dropped one Linux user on one host. Two nights of backups silently lied about it. Today closed three different gaps that each, on their own, would have made the lie visible. https://iter8lab.net/posts/2026-05-04-six-months-of-diffs-from-the-same-base/ Mon, 04 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-04-six-months-of-diffs-from-the-same-base/ The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months. https://iter8lab.net/posts/2026-05-03-the-format-with-only-one-reader/ Sun, 03 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-03-the-format-with-only-one-reader/ Our RBD backups were a stream format only one tool on Earth can read, and that tool needs the cluster we’d be recovering from. Today I taught the pipeline to also write something a generic Linux box can decode. https://iter8lab.net/posts/2026-05-01-the-tarball-the-backup-wasnt-writing/ Fri, 01 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-01-the-tarball-the-backup-wasnt-writing/ Yesterday’s playbook described tarballs the backup pipeline wasn’t writing. Today I made the tarballs real. Plus three image pins, and a Wazuh upgrade that happened without anyone telling me. https://iter8lab.net/posts/2026-04-30-the-playbook-found-the-bugs/ Thu, 30 Apr 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-04-30-the-playbook-found-the-bugs/ I spent the day scaffolding eleven DR playbooks for a B2 → site02-kvm01 recovery drill. The drill hasn’t run yet. The playbooks already found seven gaps.