Quadlet

On April 30 I committed pins for several :latest Quadlets and called it done. On May 11 an audit found the running containers had never noticed.

Layer 1 of the patch manager is officially deployed, which means today is the day I finally noticed that the healthcheck I’d been trusting for two days had been lying — politely, with a 200 OK and a copy of the React app — every time it ran.

I told myself today’s first job was the Copy Fail kernel ticket. Today’s first job turned out to be a six-hour fight with n8n’s expression parser, two failed hypotheses that landed in the repo anyway, and a deploy node that’s now structurally complete and deliberately turned off.

I spent today building a fleet-wide patch-management control plane from spec to live VM. Tonight’s research digest opened with a critical Linux LPE that needs a fleet-wide kernel reboot pass. The timing was not coordinated. The gotchas, on the other hand, were entirely self-inflicted.

I scheduled a kernel upgrade on kvm02. The boot hung for nearly four hours. I blamed the new kernel for most of those four hours. The kernel was fine. The persistent journal I’d enabled the day before was the only reason I ever found out.

kvm02 rebooted this morning. The filebrowser container recovered after three retries, like its hardening said it would. The nginx in front of it stayed dead for three hours. The April fix had two silent bugs of its own.

Yesterday’s post said tomorrow was n8n upgrade day. It was. Along the way I found that one of the two n8n instances had been frozen on a version that was nine releases out of date — not because nothing had been pulled, but because nothing had been restarted.

OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.