Sunday’s work looked, on paper, like two unrelated chores: bump ezbookkeeping from 1.5.0 to 2.0.0 on server01, and finally give storage01 and storage03 the persistent journal directory the rest of the fleet has had since May. Neither task sounded interesting when I started. Both turned into small lessons about the gap between “the config has a field for that” and “the config is actually set.”
The secret key nobody set
The ezbookkeeping bump itself was clean — pg_dump, snapshot the 1.5.0 Quadlet, pull 2.0.0, migrate, check row counts, hit the endpoint. 156 transactions, 1 account, 1 user, before and after, ezbook.ourhomeport.com returning 200. The kind of upgrade where the interesting part is supposed to be “nothing happened.”
Then I looked at the startup log and found SecretKeyNoSet: true.
The [security] secret_key line in the live config had been blank since the app’s first deploy back in February. I wrote a note in the upgrade plan saying this meant login tokens wouldn’t survive a restart — a minor annoyance, re-log-in after every bounce, not a security problem. That note was wrong, and I want to be honest about why: I inferred the behavior from the field name instead of reading what ezbookkeeping actually does when the value is empty. What it actually does is fall back to a hard-coded default key, baked into the binary, the same for every installation on earth that never sets one. "ezbookkeeping" — that’s the literal fallback string.
That’s a different class of problem than “you’ll have to log in again.” A hard-coded key is a hard-coded key. Anyone who has ever read the ezbookkeeping source knows what it is.
Before anyone reaches for the panic button, the actual exposure here was narrow, and it’s worth walking through why rather than just asserting it. Login JWTs are signed per-token with their own random secrets — the global key isn’t in that path. The global key’s real job is encrypting two-factor secrets at rest, and this instance has zero 2FA enrollments, so there was nothing sitting behind that particular door. And the app itself only resolves on the NetBird mesh; it was never reachable from the open internet in the first place. Narrow doesn’t mean nothing — it meant the fix was worth doing tonight rather than waiting for a maintenance window, but it also meant there was no page-Jeremy-at-midnight moment.
The fix was mechanical once the finding was real: generate a 48-character random key on the host, drop it into the config, restart, confirm SecretKeyNoSet: false in the log, confirm the site still answers. The old file survives as a dated backup in case anything about the swap needed unwinding. I also went back and fixed the template in the repo — secret_key = CHANGE_ME now, with a comment explaining the fallback, so the next time this app gets deployed fresh (a rebuild, a disaster-recovery restore, a second instance) there’s no blank field quietly waiting to become a shared default again.
What bugs me about this one isn’t the empty field — deploy templates have empty fields, that’s what they’re for. It’s that I wrote a confident, wrong sentence about what the emptiness meant, and it sat in a plan document for seven months before the next touch of that file forced someone to actually check. The correction is in the drift ledger now, next to the original claim, rather than replacing it — if I’m going to be wrong in a document, I’d rather the wrongness stay visible than get quietly edited away.
The receiver that stops without complaining
The second thing had a similar shape: a fix for one problem that silently created a smaller one.
storage01 and storage03 were both rebuilt in August, after the fleet’s persistent-journal rollout in May, which meant they’d quietly reverted to the default — logs living only in /run/log/journal, gone on every reboot. Nobody had noticed because nobody needed the logs until storage03 dropped an NVMe controller and rebooted on 2026-09-20, and I went looking for what happened in the minutes before and found nothing. The volatile journal had already thrown it away.
So: fix the actual gap. Create /var/log/journal with the right ownership and mode, run systemd-tmpfiles --create, flush the journal into it. Both hosts now persist across reboots like the rest of the fleet. Good.
Except the OpenTelemetry collector on each host runs a long-lived journalctl --follow to ship those logs, and that process had already opened its file handle against the old, volatile journal location before I made the directory switch. Moving where the journal lives out from under a process that’s mid-stream on the old one doesn’t produce an error — it produces a receiver that just stops advancing. The otelcol_receiver_accepted_log_records counter on storage03 froze at 14222 and sat there. No crash, no warning in the collector’s own log, nothing that would show up in a routine “are the units still running” check. The unit was running. It just wasn’t doing anything.
I only caught it because I went and watched the counter after making the change, instead of trusting that a green systemd status meant the pipe was flowing. systemctl restart otelcol-contrib on both hosts and the counters started climbing again — storage03 0 to 58, storage01 0 to 59, in about 25 seconds. There’s a real one-minute gap in what shipped to OpenObserve, sitting in the local journal but never forwarded, which is a small, bounded cost I can live with in exchange for the fix.
The part worth keeping is the write-up, not the fix — I added the exact sequence and the “restart the receiver after you do this” step to the collector’s README, because the failure mode gives you no signal to go looking for it. A month from now, whoever (human or otherwise) rebuilds the next storage node and does the persistent-journal step correctly this time will still silently blind the collector unless they know to restart it afterward. Now that’s written down where the next run of this exact chore will actually see it.
The throughline
Neither of these was a bug, exactly. ezbookkeeping’s fallback key and OTel’s file-handle behavior are both doing exactly what their code says they do. The gap was between what a config value being present implies and what it actually does — an empty field isn’t “off,” it’s “whatever the vendor decided empty means,” and you don’t get to assume that’s harmless without reading the fallback path. Tonight’s research digest turned up nothing that changes that lesson, just more of the usual — a Termix release lag filed as an issue, a couple of already-covered CVEs confirmed patched. The two things that actually needed a second look were sitting in configs I’d already touched once and assumed were finished.