Quadlet on iter8lab

Pinning What I Thought Was Already Pinned

Mon, 11 May 2026 00:00:00 +0000

On April 30 I committed pins for several :latest Quadlets and called it done. On May 11 an audit found the running containers had never noticed.

The 200 That Meant Nothing

Sat, 09 May 2026 00:00:00 +0000

Layer 1 of the patch manager is officially deployed, which means today is the day I finally noticed that the healthcheck I’d been trusting for two days had been lying — politely, with a 200 OK and a copy of the React app — every time it ran.

The Tenth Deploy Stayed Disabled

Fri, 08 May 2026 00:00:00 +0000

I told myself today’s first job was the Copy Fail kernel ticket. Today’s first job turned out to be a six-hour fight with n8n’s expression parser, two failed hypotheses that landed in the repo anyway, and a deploy node that’s now structurally complete and deliberately turned off.

Building the Patch Manager on Copy Fail Day

Thu, 07 May 2026 00:00:00 +0000

I spent today building a fleet-wide patch-management control plane from spec to live VM. Tonight’s research digest opened with a critical Linux LPE that needs a fleet-wide kernel reboot pass. The timing was not coordinated. The gotchas, on the other hand, were entirely self-inflicted.

It Wasn't the Kernel

Wed, 06 May 2026 00:00:00 +0000

I scheduled a kernel upgrade on kvm02. The boot hung for nearly four hours. I blamed the new kernel for most of those four hours. The kernel was fine. The persistent journal I’d enabled the day before was the only reason I ever found out.

The Defense That Was Never Engaged

Tue, 05 May 2026 00:00:00 +0000

kvm02 rebooted this morning. The filebrowser container recovered after three retries, like its hardening said it would. The nginx in front of it stayed dead for three hours. The April fix had two silent bugs of its own.

Latest Was Two Months Ago

Wed, 29 Apr 2026 00:00:00 +0000

Yesterday’s post said tomorrow was n8n upgrade day. It was. Along the way I found that one of the two n8n instances had been frozen on a version that was nine releases out of date — not because nothing had been pulled, but because nothing had been restarted.

The README Was Lying

Mon, 27 Apr 2026 00:00:00 +0000

OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.