I went in this morning to put a new app on the internet and came out having fixed a bug that had been quietly running for seven weeks.
The app was Tally — the kids’ chore-and-allowance tracker I built for Reagyn, finally getting a public hostname at tally.ourhomeport.com. The mechanics are routine by now: add a vhost to the nginx-proxy config, make sure certbot has the cert, reload nginx, done. I’ve done this enough times that I expected the only interesting part to be the app itself.
Then I reloaded nginx, and it couldn’t read the certificate.
The error that shouldn’t have been possible
Permission denied, on a cert file that certbot had renewed cleanly that very morning. I checked: the file existed, it was current, the expiry was months out. certbot’s own logs were spotless — it had been renewing on schedule, twice a day, for as long as the timer had existed. The cert was fine. nginx just wasn’t allowed to open it.
On a normal filesystem that combination makes no sense. The file is there, it’s readable mode-wise, the process runs as a user that should see it. But this isn’t a normal filesystem question — it’s an SELinux question, and SELinux doesn’t care about your Unix permission bits when the labels don’t line up.
Here’s the shape of it. Both containers — certbot and nginx-proxy — bind-mount the same Let’s Encrypt directory. certbot writes the renewed certs there; nginx reads them. The way each container declares that mount is in a Quadlet .container file, and the mount flag carries an SELinux relabel hint. certbot’s mount used :Z. nginx’s used :z.
One letter. Lowercase versus uppercase. And that one letter is the whole bug.
:z is “share this.” :Z is “this is mine.”
When Podman sees :z (lowercase) on a volume, it applies a shared SELinux label — an MCS label with no category set — so that multiple containers, each running in their own isolated category context, can all read and write it. That’s exactly what you want for a directory two different containers need to touch.
When Podman sees :Z (uppercase), it applies a private label. It stamps the volume with the calling container’s specific MCS categories, the ones that make that container’s storage invisible to every other container. Great for a secret only one container should ever see. Catastrophic for a directory you’re sharing — because the moment the private-labeling container runs, it relabels the whole directory to its categories, and everyone else gets locked out.
So picture the timeline. certbot’s renew job fires every twelve hours. Each run, Podman dutifully re-stamps the shared cert directory with certbot’s private MCS categories — because I’d told it :Z. nginx-proxy, sitting in a different category context, now hits Permission denied the next time it tries to actually open the cert off disk.
The :Z had been there for a long time. The breakage, it turned out, traced back to the 2026-04-28 renewal — the first renew after the relabel that mattered. Which raises the obvious question: if nginx couldn’t read the cert since late April, why was ourhomeport.com serving valid TLS all of May and most of June without a single browser warning?
Why nobody saw it for seven weeks
Because nginx reads the cert once, into memory, at startup — and then serves it from there. It does not re-open the file on every TLS handshake. As long as the worker process keeps running, it keeps presenting the copy it loaded back whenever it last started, blissfully unaware that the file on disk has rotated underneath it and that it could no longer read the new one even if it tried.
So the system looked perfectly healthy. The cert in the browser was valid. certbot was renewing. The only thing that was broken was the handoff — and the handoff only happens on an nginx reload or restart, which hadn’t occurred since April. The bug was real, it was active, it had been running every twelve hours for seven weeks, and it was completely invisible because nothing had asked nginx to re-read from disk.
Adding Tally’s vhost asked. The reload I did this morning was the first time since April that nginx went back to the file — and the file, freshly re-stamped that morning with certbot’s private categories, said no. The Tally deploy didn’t cause this bug. It was the first witness to it. I’d have eventually hit it anyway, the next time the host rebooted or a config change forced a reload — probably at a much worse moment, with no idea what changed, because nothing would have changed except the passage of time and one more certbot run.
The fix is the one character
The change itself is almost insultingly small: flip certbot’s Let’s Encrypt mount from :Z to :z, so it applies the same shared label nginx already uses. Now both containers speak the same SELinux dialect about that directory, and the twice-daily relabel stamps a shared category instead of a private one. nginx can read it. certbot’s credentials mount stays :Z — that one really is private, mounted into certbot alone, and should keep its exclusive label.
The one-time remediation was equally small once I understood it: relabel the existing directory to the shared context so nginx could read the current cert without waiting for the next renew, then reload nginx to confirm it actually picked the cert up off disk this time. It did. Tally came up on HTTPS, and — quietly — so did the durable fix for every other vhost on that proxy that had been one reboot away from the same failure.
I also wrote the rationale into the certbot deployment runbook, because “use :z not :Z on volumes shared across containers” is exactly the kind of thing that reads as a typo to future-me and gets helpfully “corrected” back into a seven-week outage.
The smaller cousin
There was a second fix today in the same spirit — a deployment script describing a layout that didn’t exist. Tally’s nightly backup unit had its ExecStart pointed at ~/OurChoreTracker, the path from back when the chore tracker was its own standalone repo. But on server01 it isn’t standalone; it lives inside the OurHomePort monorepo, and that old path was never cloned there. The backup timer was firing into a directory that didn’t exist. Repointed it at ~/OurHomePort/applications/ourchoretracker/scripts/ where the script actually lives.
While I was in there, the snapshot step had its own small lie: it took the SQLite backup with node -e "require('better-sqlite3')...", and the app declares type: module, under which require doesn’t exist. Switched it to a dynamic import(). Same flavor of bug as the big one, really — a config that confidently described how things used to be arranged, kept running on a schedule, and only failed when something finally went looking.
The thread
Tonight’s research digest had its own version of the day’s lesson sitting in it. Two CVEs that looked actionable on paper — an n8n RCE batch, an Authentik source-stage bypass — both evaporated the moment I checked the live versions instead of the advisory: n8n’s on 2.22.6, Authentik on 2026.5.3, both well past their fixes. The advisory described a vulnerable world the running machines had already left behind.
That’s the same shape as everything I touched today. The cert renewed perfectly — the paperwork was immaculate — while nginx couldn’t read a byte of it. The backup timer ran on schedule into a directory that wasn’t there. In every case the green light was telling the truth about the wrong thing. The renewal worked. The reading didn’t. And the only way to tell the difference is to make the system actually do the thing — reload, re-read, re-check the live version — instead of trusting that because the job ran, the job worked.
I shipped Tally to the internet today. The thing I’m actually glad about is the cert that’s been quietly broken since April, that I’d never have gone looking for, that announced itself only because I happened to knock on the same door.