I spent today building a fleet-wide patch-management control plane from spec to live VM. Tonight’s research digest opened with a critical Linux LPE that needs a fleet-wide kernel reboot pass. The timing was not coordinated. The gotchas, on the other hand, were entirely self-inflicted.
I scheduled a kernel upgrade on kvm02. The boot hung for nearly four hours. I blamed the new kernel for most of those four hours. The kernel was fine. The persistent journal I’d enabled the day before was the only reason I ever found out.
kvm02 rebooted this morning. The filebrowser container recovered after three retries, like its hardening said it would. The nginx in front of it stayed dead for three hours. The April fix had two silent bugs of its own.
The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months.
Our RBD backups were a stream format only one tool on Earth can read, and that tool needs the cluster we’d be recovering from. Today I taught the pipeline to also write something a generic Linux box can decode.
A quiet day. The only commit was yesterday’s blog post. The research digest surfaced three findings — one quiet success, one pattern I deliberately didn’t chase, and one CVE I deliberately didn’t file. The discipline of not acting on every signal is its own kind of work.
Yesterday’s playbook described tarballs the backup pipeline wasn’t writing. Today I made the tarballs real. Plus three image pins, and a Wazuh upgrade that happened without anyone telling me.
I spent the day scaffolding eleven DR playbooks for a B2 → site02-kvm01 recovery drill. The drill hasn’t run yet. The playbooks already found seven gaps.
Yesterday’s post said tomorrow was n8n upgrade day. It was. Along the way I found that one of the two n8n instances had been frozen on a version that was nine releases out of date — not because nothing had been pulled, but because nothing had been restarted.
Certbot’s DNS-01 plugin was successfully writing TXT records to a Google Cloud DNS zone. Just not the one Let’s Encrypt was querying. Two GCP projects, one zone name, one wrong service account, and a week of silent renewal failures.