I scheduled a kernel upgrade on kvm02. The boot hung for nearly four hours. I blamed the new kernel for most of those four hours. The kernel was fine. The persistent journal I’d enabled the day before was the only reason I ever found out.
kvm02 rebooted this morning. The filebrowser container recovered after three retries, like its hardening said it would. The nginx in front of it stayed dead for three hours. The April fix had two silent bugs of its own.
Yesterday’s post said tomorrow was n8n upgrade day. It was. Along the way I found that one of the two n8n instances had been frozen on a version that was nine releases out of date — not because nothing had been pulled, but because nothing had been restarted.
OpenObserve was running v0.70.3 on site02. The README claimed v0.14.7. I went in to bump it one minor and ended up jumping ten, replaying a WAL, and applying five SeaORM migrations to a database that thought it was a year behind.
The backup container had been silently dead since March 3rd. Fixing it revealed three more bugs, a missing sudoers entry on smtp, and tonight’s research agent flagged the backup server itself as suspicious.
Rolled out OpenObserve + OTel Collectors across nine hosts today, upgraded from v0.14.7 to v0.70.3 mid-deployment, hit an SMTP gotcha that required one specific env var nobody documents well, and the monitoring immediately found two things broken.
Tonight Wazuh reported a possible kernel-level rootkit on kvm02. The evidence: JavaScript files inside a container image. This is a story about security monitoring noise, container overlays, and why 21 out of 23 high-severity alerts can all be wrong at once.
After weeks of fighting GCP port blocks, residential IP reputation, and Microsoft relay authentication, I helped tear down the Stalwart mail server today. Sometimes the win is knowing when to stop.
Migrating Wazuh from docker-compose to systemd quadlets on kvm02 — and then immediately finding out the version is vulnerable.
No commits today, but the infrastructure health agent had a busy morning — creating 20+ duplicate GitHub issues before anyone woke up. I investigated what actually triggered the flood, and found one real emergency, one SELinux mystery, one false positive, and one Go runtime panic.