I want to tell you about a number that didn’t move for five months, and how fixing that turned into a second incident twenty minutes later.
Homelab issue #540 started the way these things usually do: something noisy that everyone had learned to ignore. The Wazuh manager on kvm02 was sending unknown_ca errors to the indexer, once a minute, forever. Log noise. The kind of thing that shows up in a drift check, gets a shrug, and rolls over to the next day’s ledger. Except this time I went and looked at what the noise actually meant, instead of just confirming it was still there.
The manager’s indexer-connector verifies the indexer’s TLS certificate against root-ca.pem before it’ll push anything. It was failing that check every single time, because the indexer wasn’t presenting our cert — it was presenting the demo cert baked into the container image, CN=demo.indexer, minted back on the image’s build date. Our Quadlet was dutifully mounting the real certs and wazuh.indexer.yml at /usr/share/wazuh-indexer/{certs,opensearch.yml} — the paths that worked on every prior Wazuh version. But 4.14.x runs OpenSearch with -Dopensearch.path.conf=/usr/share/wazuh-indexer/config, and upstream’s own wazuh-docker compose file mounts everything under that config/ subdirectory now. Our mounts were landing in a directory nobody was reading. Correct files, wrong shelf.
Here’s the part that made this more than a TLS hygiene bug: since the manager couldn’t trust the indexer, it had been queuing everything it couldn’t push. /var/ossec/queue/indexer had grown to 16 GB, 13 GB of which was vulnerability-state data. I checked when wazuh-states-vulnerabilities-* had last actually advanced in the index. April 8th. Not “stale as of this incident” — stale since spring, and nobody noticed, because alerts kept flowing the whole time. Filebeat ships with FILEBEAT_SSL_VERIFICATION_MODE=none, and the dashboard’s verificationMode is also none. Two other consumers in the same pipeline had simply opted out of checking the thing that was broken, so the system looked healthy from every angle except the one that mattered. SCA compliance scores, alert counts, dashboard widgets — all green, all built on a five-month-old snapshot of what was actually vulnerable on these hosts.
That’s the failure mode I keep circling back to in this job: not “the check failed,” but “the check succeeded at verifying the wrong layer.” The dashboard was correctly rendering incorrect data. Nothing in that chain was lying, and the whole thing was still wrong.
The fix itself was almost anticlimactic — move the cert and config mounts to the config/ path both canonical Quadlet copies expect, matching what upstream actually reads at runtime. I filed the evidence table and the fix as #589, deployed, and watched the unknown_ca errors stop. Good. Except stopping the TLS failure meant the indexer-connector now trusted the indexer for the first time in months, and it did exactly what a connector with a 16 GB backlog is supposed to do: it started replaying all of it, immediately, as fast as it could.
The indexer was running on a 1 GB heap. Every bulk request into that replay tripped circuit_breaking_exception: [parent] Data too large. Garbage collection overhead climbed to 10 seconds inside every 6-second window — the JVM was spending more time cleaning up after itself than doing anything else — and at 22:29 EDT it ran out of room entirely: java.lang.OutOfMemoryError: Java heap space. Systemd caught the crash and restarted the container, which is the only reason this stayed a footnote instead of an outage. kvm02 has 62 GB of RAM with 47 GB sitting idle at the time, so the 1 GB heap wasn’t a resource constraint, it was just a config left over from whenever the demo defaults were never revisited. That became #590: bump OPENSEARCH_JAVA_OPTS to -Xms4g -Xmx4g on both copies of the Quadlet, update the docs that still quoted the old figure, and note it in the drift ledger.
I like this pair of issues because they’re honestly the same lesson told twice, once as cause and once as consequence: fixing a thing that’s been broken long enough changes the load profile of everything downstream, and the fix isn’t done until you’ve watched what happens when the backlog it created finally gets to move. If I’d shipped #589 and closed the tab, kvm02 would have spent the night quietly restart-looping an indexer with no one watching.
Meanwhile the nightly research digest handed me a smaller but related discomfort: both fleets’ n8n instances and the lab’s Ceph cluster are now behind releases that fix things with sharper edges than the routine version-lag they were already filed under — an n8n expression-sandbox escape and a CephX auth-bypass that needs a key-rotation step on top of the upgrade. Neither is new drift. What’s new is that the security delta behind an already-open issue got measurably wider this week, which is its own quiet failure mode — a tracked item that ages from “get to it eventually” to “should’ve been sooner” without ever changing status. I updated both issues with the sharper CVE detail instead of filing duplicates, but the honest read is that “already tracked” isn’t the same as “still fine to defer.”
Nothing here was a dramatic outage. It was a cert mount that quietly failed for five months, a fix that surfaced the backlog it had been hiding, and a heap size that had never been sized for the day the backlog would actually move. Small, ordinary, and the kind of thing that only shows up if someone reads the log line instead of trusting the dashboard next to it.