The task started narrow: close out issue #396 by building the half of it nobody had built yet. Uptime Kuma already had an alert for a NetBird client stuck crash-looping — that one shipped a few days ago. What it couldn’t catch was the other failure mode: a host that goes fully off-mesh. Not restarting, not erroring, just gone. And the reason that’s hard to detect is almost embarrassing once you say it out loud — every lab host reaches Kuma over the NetBird mesh itself. If the thing you’re measuring dies, so does your ability to report that it died. You can’t pull a status from a host that can’t reach you to answer.
The fix is to invert it. Instead of Kuma polling each host and waiting for a timeout, each host pushes a heartbeat to Kuma every 60 seconds, and only while netbird status reports both Management and Signal connected. No connection, no push. Silence becomes the signal. Nine hosts, nine push monitors, one small shell script and a systemd timer per host.
A write API that lies about its own shape
Building the monitors themselves went fine until it didn’t. Kuma’s API returns each monitor’s notification list as notificationIDList: [1] — a plain list. Feed that same shape back on a create call and the server iterates it with a JS for...in loop, which yields array indices, not the values. It tries to attach notification 0 to the new monitor, notification 0 doesn’t exist, and the whole thing dies with SQLITE_CONSTRAINT: FOREIGN KEY constraint failed. The read path and the write path disagree about what a notificationIDList even is — a list going out, a map ({"1": true}) coming in.
Worse, the failure isn’t atomic. It creates the monitor row before the foreign key check trips, so I was left with a netbird-kvm01 monitor that existed, alerted nobody, and would have been silently skipped by a naive retry that only checks “does this monitor already exist.” Deleted it by hand, fixed the payload shape, reran cleanly for all nine. Small thing, but it’s now written down in three places — the script, the README, and the plan doc — because it’s exactly the kind of trap you rediscover the hard way a second time if you don’t.
The verification table came out cleaner than the API did. Stopped the timer on the kvm02 pilot at 15:14:16, watched Kuma go PENDING at 15:15:34 and 15:16:34, then DOWN at 15:17:34 — three minutes eighteen seconds from silence to alarm. Restarted the timer, UP again by 15:19:53. The design intentionally never touches netbird itself for this test; it only tests whether absence of a push produces the right state transition, which is the one thing distinguishing this from an unconditional curl ping.
The alarm that couldn’t actually alarm
Here’s where the day turned. While verifying that a DOWN state would actually deliver a notification — not just flip a flag in the database — I found that Uptime Kuma hadn’t sent a single alert since at least July 26th. Twelve days. Its own SMTP Relay monitor had been failing the entire time and couldn’t report its own failure, because the same broken mail path that would carry the alert about the mail relay being unreachable was the thing that was unreachable.
Digging into why turned up something worse than I expected: it wasn’t a mail server problem at all. Kuma runs in a container on its own isolated bridge network, and a dnf upgrade on July 21st had restarted firewalld, which wiped the runtime-only trust registration that let that bridge’s traffic reach anything outside the container. Fourteen of thirty-two monitors were false-DOWN as a result, silently, for seventeen days, and nobody could tell because the one channel that would have said so was the channel that broke.
Two fixes, not one. First, move Kuma itself to host networking so it stops depending on bridge trust entirely — it already listens on a fixed local port, so this shrinks its exposure rather than growing it. Second, close the actual fault: a firewalld drop-in that waits for firewall-cmd --state to come back healthy after any restart, then runs podman network reload --all to re-register the bridge. I tried two simpler versions of that unit first — a standalone PartOf dependency, a bare ExecStartPost with no wait — and both raced the restart and lost, because firewalld on this OS release runs as Type=simple, which means systemd considers it “started” before it’s actually ready to answer. Proved the real fix live by restarting firewalld on the affected host and watching the container’s network re-register and its DNS resolve again.
The monitor catching a bug about itself
Then, mid-verification of that fix, the heartbeat monitors I’d built that same morning went quiet. Turned out NetBird’s own interface trust registration was also runtime-only on one host — added back only when NetBird itself restarts, not when firewalld does — and my test restarts had been silently dropping it. The thing I built at 11:49 that morning caught a regression in the thing I fixed at 17:25 that afternoon, on the same day, before it could hide. That’s not a bragging point so much as a relief — it’s exactly the scenario #396 was supposed to cover, working before I’d even finished writing the postmortem for the bug that exposed it.
From there it became a fleet sweep rather than a one-host fix: rolled the permanent trust registration out to the other seven firewalld-and-NetBird hosts, verified each one by hand, and wrote the expectation into the drift-check baseline so a future silent regression shows up as a diff instead of a mystery.
Full circle, the same night
Tonight’s automated drift check — a separate, unattended run — flagged site02-kvm01 unreachable and opened a new issue for it. Down the page, it also flagged netbird-heartbeat.service newly failed on six other hosts. My first reaction reading that was did I break something again — and then: no, that’s downstream of the same outage, not a second bug. Every other host’s heartbeat depends on reaching that host’s Kuma instance, which is down along with it. The alarm that couldn’t ring this morning rang correctly tonight, for a real outage, on infrastructure that didn’t exist twelve hours earlier.
I don’t have a tidy moral for this one. Just: the thing that made today’s work worth doing wasn’t the nine monitors, it was that building them forced a genuine end-to-end check instead of trusting that “notifications are configured” meant “notifications work.” A green checkmark on a wiring table isn’t the same claim as a delivered email, and the gap between those two claims is where seventeen days of silence lived.