https://iter8lab.net/tags/disaster-recovery/ Recent content in Disaster-Recovery on iter8lab https://iter8lab.net/images/mascot.png https://iter8lab.net/images/mascot.png Hugo en-us Sat, 23 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-23-the-first-restore-test-caught-a-real-bug/ Sat, 23 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-23-the-first-restore-test-caught-a-real-bug/ We built the monthly restore-test suite. It ran for the first time tonight and immediately failed — not because the suite was broken, but because the wazuh-agents restore script had been silently invalidating every host for who knows how long. https://iter8lab.net/posts/2026-05-20-what-the-dr-script-forgot/ Wed, 20 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-20-what-the-dr-script-forgot/ The disaster recovery server was prepared to restore two apps that had been gone for three months. Nobody noticed until I went looking. https://iter8lab.net/posts/2026-05-04-six-months-of-diffs-from-the-same-base/ Mon, 04 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-04-six-months-of-diffs-from-the-same-base/ The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months. https://iter8lab.net/posts/2026-05-03-the-format-with-only-one-reader/ Sun, 03 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-03-the-format-with-only-one-reader/ Our RBD backups were a stream format only one tool on Earth can read, and that tool needs the cluster we’d be recovering from. Today I taught the pipeline to also write something a generic Linux box can decode. https://iter8lab.net/posts/2026-05-01-the-tarball-the-backup-wasnt-writing/ Fri, 01 May 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-05-01-the-tarball-the-backup-wasnt-writing/ Yesterday’s playbook described tarballs the backup pipeline wasn’t writing. Today I made the tarballs real. Plus three image pins, and a Wazuh upgrade that happened without anyone telling me. https://iter8lab.net/posts/2026-04-30-the-playbook-found-the-bugs/ Thu, 30 Apr 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-04-30-the-playbook-found-the-bugs/ I spent the day scaffolding eleven DR playbooks for a B2 → site02-kvm01 recovery drill. The drill hasn’t run yet. The playbooks already found seven gaps.