An infrastructure engineer and his AI build, break, and document a homelab — one iteration at a time.
The Reboot That Finally Left Evidence
A fourth silent smtp reboot finally came with a vmcore attached, and the mechanism behind it turned out to be a Ceph mon flap wearing a kernel watchdog costume.
The Dependency That Was Already Dead
A routine Authentik version bump turned into deleting a container instead of patching it, once I noticed nothing had talked to it in months.
The Version String That Lied
A day spent closing out version-drift tickets turned into a lesson about trusting --version output too literally.
The Backup That Thought It Was Late
A backup scheduler compared local calendar days against SQLite’s UTC wall clock, and quietly decided every night was a missed night.
The Bill That Vanished for Four Days
A Backblaze charge went missing from the bills list, and chasing it down turned into a lesson about anchoring dates to the wrong clock — twice in one afternoon.
The NetBird Bump We Skipped a Version For
Bumping NetBird to 0.78.1 across twelve hosts in two repos, and why 0.78.0 got skipped entirely.
A Ceph CVE Fix Broke Libvirt, and Nobody Noticed for Two Weeks
Patching four Ceph CVEs quietly broke every libvirt RBD storage pool in the lab — the fix was to stop needing them.
Receipts in the Mail
Teaching Ledgerline to read receipts out of an inbox, and the two-Run-now-attempts bug that briefly blanked the settings page.
Five Months of Silence
A cert-verification gap let 13 GB of vulnerability data sit frozen since April; fixing it triggered a second failure I hadn’t budgeted for.
The Upgrade That Touched Everything at Once
A NetBird fleet upgrade that spanned two separate infrastructures in one sitting, plus an offline filesystem repair I had to hand off instead of run myself.