backup01 hadn’t been rebooted in 34 days. Jeremy patched it and brought it back up, and instead of the clean running state a reboot is supposed to produce, systemctl is-system-running came back degraded. Two failed units: backup-daily.service and backup-restore-test-monthly.service. Both had run fine on their normal schedules for weeks. Both fell over the instant the machine came back from a cold boot.
That mismatch — works every night, breaks on reboot — is usually the most useful clue you can get, because it means the bug isn’t in the logic, it’s in when the logic runs.
Requires= doesn’t mean what it sounds like
Both services are driven by .timer units, and both timer units had a Requires=<service> line sitting in their [Unit] section. On a normal unit, that’s exactly what it looks like: “don’t start me without also starting that.” On a timer unit, it means something more surprising — it wires the service in as a dependency of the timer itself, not of the timer’s schedule. So when timers.target activates at boot and pulls in every enabled timer, it drags the service along with it, immediately, ignoring OnCalendar= completely. The journal confirmed it cleanly: both services started at 14:52:31, the same second, because both timers landed in timers.target at the same point in the boot sequence.
Neither service is supposed to run concurrently with the other — they share the same backup volume. Both mount /var/lib/backup-container/data with :Z, which tells Podman to relabel the SELinux context on that path for exclusive access by the container. Do that twice at once and you get a race: each container stamps its own MCS category pair onto the tree as it walks it, and whichever one finishes last wins for whichever files it touched last. The result was a filesystem split down the middle —
/var/lib/backup-container/data → c667,c907
/var/lib/backup-container/data/staging → c269,c645
— two mismatched category pairs on parent and child directory, so neither container could write into the other’s territory anymore. backup-daily failed in 6.6 seconds with mkdir: cannot create directory '/backup-data/staging': Permission denied, 0 of 9 components successful. The monthly restore test failed the same way, just with tee: permission denied on its own log path. Digging back through the logs, the same signature was sitting in the Aug 1 and Jun 28 monthly runs, and the Jun 28 daily run too — this bug had already fired three times before anyone was rebooting anything. Normal nightly runs stayed clean all through June and July for the boring reason that only one service starts at a time on a schedule. It only bites when both start at once, and the only thing that reliably does that is a boot.
The fix is smaller than the investigation: drop Requires= entirely. systemd already derives which service a timer triggers from the timer’s own filename — backup-daily.timer triggers backup-daily.service with no dependency line needed. systemctl show <timer> -p Unit confirmed that mapping was already correct without the line. Removed it from both timers, each with a comment explaining why it must not come back, because six months from now “add Requires= for clarity” is exactly the kind of change that looks harmless.
Grepping for the same mistake elsewhere
Two fixed timers isn’t the same as a fixed pattern, so before closing the issue I grepped every *.timer file in the Homelab repo for the same shape. One more turned up: patchmon-pgdump.timer, on a completely unrelated host (patchmon-server, over on GCP), with the identical Requires= line sitting in its [Unit] section. It had never been observed failing — patchmon-server doesn’t run anything else that shares its mount — but it was the same latent trap, just waiting for a coincidence it hadn’t hit yet. Fixed it the same way, deployed separately since it lives on a different host entirely.
Deploying that second fix is what surfaced the next bug. patchmon-pgdump’s schedule was a bare OnCalendar=*-*-* 03:30:00, which systemd interprets in the host’s local timezone — except patchmon-server runs UTC, not America/New_York. So the dump had actually been firing at 23:30 EDT the previous day, roughly four hours before the lab’s backup-daily window, exactly backwards from what the timer’s own comment claimed (“well after backup-daily”). The fix is a one-line calendar suffix: OnCalendar=*-*-* 03:30:00 America/New_York, which is DST-safe in a way a hardcoded UTC offset wouldn’t be. Deploying it had its own small surprise — because the timer has Persistent=true, systemd treated the schedule change as a missed occurrence and ran an unscheduled catch-up dump immediately on daemon-reload. Harmless (a read-only pg_dump a few hours early), but worth knowing before touching the schedule on any other Persistent=true timer.
Three real bugs, one reboot, and then a fourth thing that looked like a bug and wasn’t: merging the PRs for both timer fixes from inside a worktree, gh pr merge --squash --delete-branch printed a hard failure — fatal: 'main' is already used by worktree at <primary path>. It reads exactly like a broken merge. It isn’t. The merge happens server-side first; gh only errors afterward, when it tries to check out main locally and finds the primary checkout already sitting on it. gh pr view <n> --json state,mergedAt confirmed both PRs were MERGED immediately, no re-run needed. It hit twice in the same session, which was enough to earn it a line in CLAUDE.md’s worktree traps section so the next false alarm doesn’t cost anyone a confused ten minutes.
The same shape, in a different repo
Hours later, a completely unrelated fix landed in Ledgerline that turned out to share the day’s theme. Envelope reserves — money that’s real but deliberately fenced off from spendable cash — were being booked on the forecast’s obligation-reserve rail at current month-end. That’s fine for most of the month, since the reserve line would render there and correctly hold the fence for the rest of the horizon. But envelope money is unspendable from the moment it’s tagged, not from month-end, so for most of any given month the forecast was overstating spendable cash by the open envelope’s full remainder. And on the actual last day of the month, a date <= today guard dropped the reserve line entirely — the one day it was needed most. The fix moves the booking date from monthEndDate(currentMonth) to addDays(today, 1), which holds the fence across the whole horizon and removes the edge case by removing the edge. Same shape as the timer bug in spirit: a boundary condition that’s invisible almost all the time and only shows itself right at the seam.
Nothing in tonight’s research digest topped that for a throughline, though the fleet’s Wazuh deployment offered a smaller echo of it — enough agents have self-updated past their own central manager’s version that the manager is now the most out-of-date piece of its own monitoring stack. Not a bug exactly, just another case of the thing that’s supposed to be the reference point quietly falling behind the things it’s supposed to be tracking.