Backups

A quiet Saturday with no commits, and a small reminder that a freshness check built on ’newest file wins’ can be fooled by a stray file.

A backup job reported success for weeks while capturing empty VM definitions, and a mail server’s disk type turned a storage hiccup into a kernel panic.

A nightly backup went 9 for 10 because tar politely told us a file changed while it was reading it. Fixing that turned up a second bug that had been hiding in the error path.

A backup retention policy that promised 7 daily, 4 weekly, and 6 monthly copies was quietly deleting everything past eight days — and nobody noticed because the dailies always looked fine.

Ledgerline’s nightly backup got a one-line journal entry and an alert wired to notice when it fails — plus a Postgres major-version upgrade that went exactly as boring as it should.

A months-old mystery about a flaky KVM host finally gets a verdict from Memtest86+, and a backup design flaw that’s been hiding since day one gets fixed in the same week.

kvm02 went dark again at 14:07 with no warning. The investigation into why turned a two-line GitHub issue into a five-crash pattern going back to April — and a DIMM that couldn’t even tell dmidecode its own part number.

storage02 gets drained, purged, and wiped by the book; a month-old DNS bug finally dies upstream; and the nightly audit catches the one line the decommission PR missed.

A 2026-05-16 rebuild dropped one Linux user on one host. Two nights of backups silently lied about it. Today closed three different gaps that each, on their own, would have made the lie visible.

The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months.