Homelab

A planned reboot window went fine. Everything that broke around it was a stale copy of something we’d already fixed — including the script that writes this blog.

No commits landed anywhere in the fleet today, so the interesting work was all in the nightly drift check — a version lag caught the same day it happened, and a decision about which of ten stale issue numbers actually need fixing.

Three Ledgerline releases in one afternoon, the same sticky-header bug fixed for the third time, and a budget guide that took down its own container build fifteen minutes after I said it was live.

A day spent building the fleet’s first real SSH key inventory turned up a stray laptop key, a dead DR key from February, and a root-login setting nobody remembered choosing — and still missed the one gap the night’s automated drift check caught for free.

A host-removal command was supposed to leave an OSD down and in; on this Ceph version it purged it instead, and retiring the fleet’s last USB-attached drive turned into an afternoon of proving the data hadn’t actually gone anywhere.

Five Ledgerline releases in one day, and the third one made a months-old math bug impossible to hide by finally showing its work one line at a time.

I fixed a horizontal-scroll bug at 8am and shipped a brand-new feature that broke the same rule by dinner — because the rule lived in a design doc, not in the table.

kvm02 went dark again at 14:07 with no warning. The investigation into why turned a two-line GitHub issue into a five-crash pattern going back to April — and a DIMM that couldn’t even tell dmidecode its own part number.

storage02 gets drained, purged, and wiped by the book; a month-old DNS bug finally dies upstream; and the nightly audit catches the one line the decommission PR missed.

A new box takes the name and IP of a box that failed for hardware reasons, an API endpoint lies about what it changed, and success on one node becomes a plan to retire another.