Yesterday’s post ended on a clean note. After a full day of dashboards, config-as-code, channel re-planning, and pinning every IoT device to its nearest strong AP, I wrote: “The FP2 stays on the garage AP. The cameras hold.” I meant it. The band looked better, the top-flappers list had cleared out, and I closed the laptop feeling like the 2.4 GHz problem was a workflow now instead of a mystery.
It was a good story. It was also wrong about the one device the whole investigation started with.
The number that didn’t move
Here’s the thing about building a dashboard before you fix something: it keeps telling you the truth even after you’ve decided you’re done. I came back to the wifi-client-disconnects view in OpenObserve to confirm the win, ran the per-device breakdown for the last 24 hours, and the garage Aqara FP2 — the mmWave presence sensor I’d loosely called “the doorbell” yesterday — was sitting at the top of the list. Not reduced. Not tamed. Eighty-one percent of every 2.4 GHz disconnect on the network. Roughly 658 drops on the 16th alone. It was reassociating every one to two minutes, all day, like a metronome.
The channel re-plan had genuinely helped the network. airthings-view disconnects were down 83%. The cameras were off the leaderboard entirely. Moving the garage AP off channel 11 — where it could hear forty-nine other access points — onto channel 6, where it could hear twelve, cut the airtime contention exactly as intended. Every device near that AP got quieter.
Except the FP2, which did not care.
That’s the trap in RF work, and I walked right into it: I fixed the environment and assumed I’d fixed the device, because the environment fix was the satisfying, hard-won one. The dashboard didn’t let me keep that assumption. One device, 81%, every two minutes — that is not the signature of a noisy channel. A noisy channel hurts everyone a little. This was one client hurting alone.
Following the wrong field
Before I could conclude that, I had to get past two pieces of misleading data, and both are worth writing down because they’d fool me again.
First, the channel field lied. The FP2’s disconnect events logged it on channels 1, 6, and 11 — under an AP that was hard-pinned to channel 6 and physically incapable of being on 1 or 11. That’s nonsense, and if I’d trusted unifi_wifi_channel I’d have spun off chasing a phantom band-steering bug. The field that actually holds up is unifi_last_connected_ap — which AP it dropped from, not what channel the client claims it was on. I wrote that distinction into the new troubleshooting runbook in bold, because the channel field is right often enough to be dangerous.
Second, there were two of them. The FP2 had an inert twin sharing the same hostname — same name in the controller, one flapping constantly, one silent. Grouping by hostname blended them into one confusing average. Grouping by MAC (54:ef:44:52:db:73) separated the patient from the bystander instantly. And a third red herring rode along the whole time: overnight Google Home “camera offline” alerts that looked like they belonged to this Wi-Fi mess. They didn’t. The cameras were holding six-hour-plus uptimes the entire night. Those were brief cloud-side blips, an entirely different layer of the stack waving its arms for attention while I was busy with RF.
It wasn’t the radio
Once the FP2 was isolated, the cause fell out fast. The flapping started — to the hour — when the sensor took a firmware update on the 16th. A clean step-up, no failed flash. And here’s the part that surprised me: a power-cycle did nothing. Pulling power and bringing it back left it flapping exactly as before. That ruled out a transient, and it ruled out RF: no channel, no pin, no min-data-rate setting makes a device disconnect from itself every two minutes through a reboot.
The fix was a factory reset. Ten quick clicks on the recessed button by the USB-C port, then re-add it in the Aqara Home app. I verified across two settled windows — once in the office, once back in the garage — and both came back zero disconnects, continuous 20-minute-plus associations, signal and data rates fully restored. Dead flat. The metronome stopped.
The interesting bit is why a reset worked when a power-cycle didn’t, and it’s a clean piece of reasoning: a factory reset does not downgrade firmware. The device came back on the same new image — and behaved. So the bad state wasn’t the firmware image; it was corrupted runtime state left behind by the update, the kind of thing a reboot reloads intact but a full wipe clears. Not RF, not the firmware version, not the hardware. Just a device that came out of an update with a scrambled brain and needed to be told to forget everything it thought it knew. (The MAC survives a reset, so the Garage_AP pin I’d set yesterday held — that part of the work did stick.)
Two documents that no longer matched reality
So today’s commits weren’t features. They were corrections. The UniFi README had a stale line claiming the channel-6 move was the FP2 fix; I rewrote it to say what’s actually true — the move cut the garage’s neighbor count and airtime but did not stop the sensor — and added a root-cause section with the factory-reset playbook for the next Aqara device that flakes after an update. Alongside it, a new read-only troubleshooting runbook capturing the recipes that cracked the case: the distroless-container credential read (OpenObserve has no shell to exec into), the microsecond time fields, the trust-the-AP-not-the-channel rule, the settled-window method for confirming a fix actually held.
And then, in the same spirit, I fixed a second drifted document on the other side of the lab. A drive swap on the Ceph storage node sent me into the USB-OSD runbook, where I found a section still insisting storage01 was kernel-pinned at 5.14.0-570.30.1 — a pin that was actually lifted almost a month ago, on May 19th, when the Rocky 9.7 kernel was validated free of the xHCI crash that prompted the pin in the first place. True for weeks, written down nowhere. I led the section with current reality and demoted the pin to historical context. The swap itself also taught the runbook two new failure modes worth codifying: a hotplug on the shared USB controller bounced a marginal bridge and crashed the OSD riding on it, and a re-enumeration left that OSD’s device-mapper table pointing at a dead /dev/sd*, producing BlueStore I/O errors until a vgchange cycle and a ceph-volume lvm activate brought it back.
There’s a theme I didn’t go looking for. Both of today’s commits were the same act: reconciling a written-down belief with what the system was actually doing. The README said the channel move fixed the sensor. The runbook said the host was pinned. Yesterday’s blog post said the doorbell stays put. All three were confident, all three were stale, and the cluster and the controller did not care what any of them claimed. The dashboard outlasted my victory lap, the swap outlasted the pin note, and the honest move both times was to go back and edit the thing I’d written into agreement with the thing that was true.
Sidebar, from tonight’s research sweep, and uncomfortably on-theme: Ceph cut 19.2.4 Squid, patching a silent OSD data-corruption bug in 19.2.0 through 19.2.3 that triggers when bluestore_elastic_shared_blobs is enabled — which is the default. I checked, and both OSDs in the cluster I was just swapping a drive in are running 19.2.3 with the flag on. Exposed, quietly, the whole time. Filed it as Homelab #294 and bumped it ahead of the larger Tentacle upgrade planning, because “silent” plus “data corruption” plus “default on” is the rare combination that earns the front of the line. Another belief — “the storage is fine” — that wanted reconciling with reality before I trusted it again.