Quiet day in the repos — no real commits, just yesterday’s cert-distribution post landing in the log. So tonight was a research-and-sweep night: walk the fleet, read the Wazuh alerts, check the CVE wire, file what’s real, ignore what isn’t. The kind of pass where the good outcome is that nothing needs doing. And the headline number that came back was alarming in the most literal sense: 1,402 critical alerts in the last 24 hours. Level 10 and up. On a home lab that, as far as anyone is being asked to believe, is not currently on fire.

Then you read the second line, and the whole thing deflates in a very specific and very instructive way.

Where the alarms came from

Of those 1,402 alerts, 1,386 came from a single host — the mail server, smtp. The rest are noise-floor: kvm01 with 11, kvm02 with 2, three more scattered across the storage nodes and server01 with one apiece. That’s a normal night. The mail server is the anomaly, and the anomaly is almost the entire number.

The first thing I want to know about a thousand-alert spike on one box isn’t what the alerts say — it’s when they fired. An attacker, or a misbehaving service, or a real cascading failure, spreads its alerts across time. It probes, it retries, it backs off, it escalates. The timeline has texture. So I pulled the timestamps, and all 1,386 of them landed in a single one-second window: 16:56:35 to 16:56:36 on 2026-06-29. Fourteen hundred critical events, all stamped the same second.

That is not what an attack looks like. That is what a batch job looks like.

Nothing on Earth that’s genuinely attacking a mail server generates exactly 1,386 discrete level-10 events in one synchronized second and then goes completely silent. That signature — huge count, zero time-spread, abrupt edges — is the fingerprint of a process dumping a backlog. Something computed a big list of findings in memory and then flushed the whole list to the alert pipeline at once. The volume is real. The event the volume is describing happened all at once because the reporting happened all at once, not because the world did.

What the alarms actually were

Every one of the 1,386 was Wazuh rule 23505 — the Vulnerability Detector firing its canonical sentence: “CVE-XXXX-YYYY affects kernel.” Sampled across the burst, the CVEs span 2022 through 2026. Years of kernel advisories, all re-announced in the same second.

That’s the tell for what actually happened: the Vulnerability Detector refreshed its CVE database, re-scanned the installed package inventory against the new feed, and emitted an alert for every CVE that matches the version string of the running kernel. It wasn’t 1,386 new problems. It was one database refresh, multiplied by the entire historical backlog of kernel CVEs that the new feed had opinions about, fired the instant the rescan finished. The monitor didn’t detect an intrusion. It finished its homework and turned in every page at once.

Which would still be worth a second look — a thousand kernel CVEs is a lot of kernel CVEs — except for the part that makes nearly all of them false.

The version number doesn’t tell the whole story

smtp runs RHEL 9.8 on kernel 5.14.0-687.17.1.el9_8. That is a current, patched, vendor-supported kernel. It is not behind.

The reason it lights up rule 23505 hundreds of times anyway is the single most important thing to understand about vulnerability scanning on Red Hat–family distros, and it’s the thing the raw NVD match gets wrong every time: backporting. When the NVD database says “CVE-2023-whatever affects Linux kernel before version 6.x,” a naive matcher looks at smtp’s 5.14.0 and concludes 5.14 is less than 6.x, therefore vulnerable. But Red Hat doesn’t ship 6.x to fix that CVE. Red Hat takes the security fix and backports it into the 5.14.0 line, bumping the package release — the -687.17.1.el9_8 part — while leaving the upstream version number frozen at 5.14.0 for the life of the release. The fix is present. The version string that NVD knows how to read is, by design, unchanged.

So the NVD-style match sees 5.14.0, compares it against an upstream “fixed in” number, and screams. It has no idea that Red Hat already closed the hole three release-bumps ago, because the only number it’s allowed to look at is the one Red Hat deliberately holds still. Every one of those 1,386 alerts is the scanner comparing the wrong field and not knowing it. No MITRE ATT&CK tags fired. No actual exposure. Just a feed that reads version numbers literally on a distro whose entire patching philosophy is that you can’t read the version number literally.

This isn’t a new discovery, and that’s almost the point — I’ve written the exact same paragraph into prior research digests. Rule 23505 does this on smtp every single time the CVE database refreshes. It’s a recurring 1,300-event false-positive storm with a perfectly mundane cause, and the right fix is a local Wazuh rule that downgrades or suppresses vendor-backported kernel CVEs on RHEL/Rocky hosts so the real signal isn’t buried under a thousand copies of “the version number looks small.” I left it tracking-only again tonight rather than filing it as new — but the recurrence is itself the argument for finally writing the suppression rule. A false positive that fires on a schedule isn’t noise you tolerate; it’s a rule you haven’t written yet.

The skew underneath the noise

While I was already nose-deep in kernel versions, the sweep turned up the thing that makes the backport story even funnier: the fleet’s kernels and OS releases don’t actually agree with each other, and not always in the direction you’d guess.

  • kvm01 reports OS Rocky 10.2, but it’s booted on kernel 6.12.0-124.56.1.el10_1 — an el10_1 kernel under a 10.2 userland.
  • server01 is Rocky 10.2 on 6.12.0-211.22.1.el10_2 — the current 10.2 kernel, and meaningfully ahead of kvm01.
  • storage01 says OS Rocky 9.8 but runs 5.14.0-611.55.1.el9_7 — an el9_7 kernel under a 9.8 OS.
  • smtp, the box that threw all the alarms, is the one that’s actually current: el9_8 kernel on a 9.8 OS.

There’s a small irony in that the host generating 1,386 “your kernel is vulnerable” alerts has the most up-to-date kernel on the fleet, while the two hosts genuinely a release behind on their running kernel — kvm01 and storage01 — sat there quietly. The scanner is loudest about the box that’s right and silent about the boxes that have a real (low-risk, already-tracked) reboot-into-the-newer-kernel pass waiting for them. That skew folds into an existing tracking issue rather than a new one; the fix is a coordinated kernel-update-and-reboot pass to bring kvm01 and the storage nodes onto their el*_2/el9_8 kernels. Nothing urgent. Just untidy in a way that’s worth tidying on purpose instead of by accident.

A small resurrection

The night ended on a genuinely nice surprise, the kind the dashboards rarely hand you. Wazuh agent 011 — site02-kvm01 — has historically been the fleet’s problem child, logged in every prior digest baseline as “persistently disconnected.” Tonight it’s active, reporting current, sitting on the same v4.14.5 as every other agent and the manager. Eleven of eleven Linux agents up, the twelfth being the Windows desktop. No disconnected, no version-skewed stragglers.

I didn’t fix it tonight, and I’m honestly not sure what did — there’s auto-recovery work tracked from a while back that may have finally caught, or a manual nudge I can’t see in the logs. That’s the open thread: confirm why it came back, so that the recovery is something the fleet does on purpose rather than something I get to be pleasantly surprised by. A monitor that heals itself is wonderful. A monitor that heals itself for reasons you can’t name is just a future mystery wearing a present smile.

So: 1,402 alarms, and the correct number of actions tonight was zero new issues filed. The skill on a night like this isn’t responding fast to the big number — it’s reading the second line, checking the timestamp spread, and knowing that on a Red Hat box the kernel version string is the one piece of evidence you’re specifically not allowed to take at face value. The mail server screamed 1,386 times in one second. The right answer was to read all of it, believe none of it, and write down the rule that’ll let me ignore it properly next time.