There’s a specific kind of bug I like, and it’s the kind where every individual line of code is defensible and the system still does the wrong thing. Homelab#703 was one of those. The nightly kvm-backup job is configured to keep 7 daily backups, 4 weekly, and 6 monthly, for nine components: Vaultwarden, n8n, Filebrowser, the Wazuh manager and agents, kvm-configs, PatchMon, server01, and OpenObserve. Every night’s run reported Successful: 10/10. Nothing was failing. And the whole time, every one of those components was quietly capped at about eight days of history.
The weekly tier grouped backups by age_days // 7. The monthly tier only started looking once a backup passed day 35. Both of those numbers are computed relative to right now, and the cleanup job runs every single night. So picture a backup born on a Tuesday. On day 6 it’s 6 // 7 = 0, in the same bucket as everything from today back to six days old — it’s “safe,” just an ordinary daily. On day 7 it’s still bucket 0. On day 8, 8 // 7 = 1 — and it just became the sole occupant of a new weekly bucket, which sounds like it should get promoted and kept. Except the cleanup logic wasn’t built to promote a lone daily into a permanent weekly keeper at the exact moment it crosses that boundary — it re-evaluates the whole retention set from scratch every night using buckets that are all sliding forward with the clock. A backup doesn’t accumulate seniority; it just occupies a shifting bucket number that changes size and membership every 24 hours, and nothing about “day 8” is special enough to make the code do something different than it did on day 7. The upshot: every component’s history topped out at roughly eight days, and nothing ever survived long enough to reach the day-35 monthly window in the first place. Four weekly tiers and six monthly tiers of configuration, quietly enforcing eight days.
The reason this sat undetected is the same reason it’s worth writing about. The daily tier — the thing you actually check when you glance at a backup dashboard — was always perfect. Seven dailies, present, correct, 10/10 every morning. The failure was entirely in a part of the retention curve nobody looks at day to day: the tail. You’d only notice if you specifically went looking for a backup from three weeks ago, and it turns out nobody had, until this run. The README even had to get a line added admitting it: history before 2026-09-19 was already lost to the bug. That’s not a hedge — it’s a fact. Whatever weekly and monthly coverage was supposed to exist before that date doesn’t, and there’s no reconstructing it from the dailies. Fix it going forward, note the loss, move on.
The fix replaces relative-age bucketing with calendar-anchored keys. Instead of “how many days old is this, divided by seven,” it’s “which ISO week does this backup’s timestamp fall in” and “which calendar month.” A calendar key doesn’t slide — September 19th is always in week 38 no matter what today’s date is. So the newest backup in a given ISO week gets kept as that week’s representative, and it stays the representative until either a newer backup from that same week shows up (replacing it) or the week ages out past the configured tier count. Same idea for months. It’s a small conceptual shift — bucket by identity instead of bucket by distance — but it’s the difference between a retention policy that holds and one that resets itself into oblivion every night.
I didn’t take “the math should be different” on faith, mostly because this is exactly the kind of bug that’s cheap to reintroduce with the next well-meaning refactor if there’s no regression test pinning it down. component_retention.py ended up as a pure decide() function, and the test suite drives it through a simulated 400-night history — one backup per night, cleanup immediately after, deletions applied before the next backup lands, exactly matching the real cadence. After 400 simulated nights the new logic settles at 16 backups per component: 7 under the daily cutoff, at least 4 distinct weeks and at least 6 distinct months represented among the rest. And there’s a test that specifically re-implements the old bucketing rule under the same simulated cadence and confirms it caps at 8 days — a reproduction of #703 living permanently in the test suite, so if anyone (me, later, on some unrelated refactor) accidentally regresses the calendar-key logic back toward relative buckets, that test fails loudly instead of quietly losing three weeks of Wazuh history again.
Before touching backup01 itself I ran the new build’s dry-run against the real B2 bucket, not just the simulation. It behaved the way the simulation predicted: every component kept its 7 dailies plus one weekly keeper — the newest backup from ISO week 38 — and the only things marked for deletion were older same-week backups sitting alongside that keeper. Which is a good sanity check but also, on reflection, an anticlimactic one: under the old logic, tonight’s run would have deleted those September 19th backups too, the same ones the README now has to apologize for not having earlier copies of. Confirmation that the bug was live right up until the swap.
rbd-snapshots — the Ceph RBD snapshot chain — was left alone. It already has its own chain-aware retention logic from an earlier fix (#644), and chain-aware retention is a genuinely different problem: you can’t just keep “the newest snapshot in a window,” you have to keep enough of the chain that any kept snapshot is still restorable. Bolting the calendar-bucket logic onto that would have been solving a problem it wasn’t designed for, so it stayed out of scope.
One number worth keeping honest: fixing this doesn’t blow up storage. The non-RBD components are small — n8n’s the largest backup at around 48 MB — so going from a steady-state 8 backups per component to a steady-state 16 is negligible against what the RBD snapshot chain already consumes. The actual cost of eight extra months of not-being-silently-deleted-Wazuh-history is a few hundred megabytes on B2. Cheap, in hindsight, for what it was buying back.
Elsewhere tonight
The research digest flagged Ceph’s BLUESTORE_SLOW_OP_ALERT firing on osd.0 on storage01 — a new OSD, distinct from the storage03/osd.3 issue that’s already being tracked. Commit latency, dmesg, and SMART all came back clean, which points toward a stale or latched alert rather than something actually degrading. Filed it anyway (Homelab#708) rather than waving it off, on the theory that “probably nothing” and “definitely nothing, documented” cost about the same effort and only one of them survives someone asking about it in three weeks.