https://iter8lab.net/tags/postmortem/ Recent content in Postmortem on iter8lab https://iter8lab.net/images/mascot.png https://iter8lab.net/images/mascot.png Hugo en-us Sun, 27 Sep 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-09-27-the-backup-retention-that-never-was/ Sun, 27 Sep 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-09-27-the-backup-retention-that-never-was/ A backup retention policy that promised 7 daily, 4 weekly, and 6 monthly copies was quietly deleting everything past eight days — and nobody noticed because the dailies always looked fine. https://iter8lab.net/posts/2026-09-21-the-key-that-was-never-set/ Mon, 21 Sep 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-09-21-the-key-that-was-never-set/ A routine version bump turned up a secret key that had defaulted to a public string since February, and a log pipeline that goes quietly blind the moment you fix the thing it was complaining about. https://iter8lab.net/posts/2026-09-19-the-dns-outage-that-took-nine-days-to-name/ Sat, 19 Sep 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-09-19-the-dns-outage-that-took-nine-days-to-name/ A resolver that looked fine from every angle except the one host that actually mattered, and the ACL that had been quietly eating its answers since September 10th. https://iter8lab.net/posts/2026-09-13-the-ram-was-guilty-all-along/ Sun, 13 Sep 2026 00:00:00 +0000 https://iter8lab.net/posts/2026-09-13-the-ram-was-guilty-all-along/ A months-old mystery about a flaky KVM host finally gets a verdict from Memtest86+, and a backup design flaw that’s been hiding since day one gets fixed in the same week.