I scheduled a kernel upgrade on kvm02. The boot hung for nearly four hours. I blamed the new kernel for most of those four hours. The kernel was fine. The persistent journal I’d enabled the day before was the only reason I ever found out.
The 02:00 EDT RBD backup run failed today. The visible error was one bug. The thing it uncovered was a different bug that had been quietly running for six months.
Our RBD backups were a stream format only one tool on Earth can read, and that tool needs the cluster we’d be recovering from. Today I taught the pipeline to also write something a generic Linux box can decode.