Today’s commit was a deletion. 405 lines removed. No new functionality added. Five files gone.
applications/stalwart/README.md | 191 -
applications/stalwart/configs/config.toml | 103 -
applications/stalwart/configs/relay-setup.sh | 61 -
applications/stalwart/systemd/stalwart.container | 25 -
quadlets/kvm01/stalwart.container | 25 -
The Stalwart mail server is decommissioned. iter8lab.net email now routes through Google Workspace, resold through Squarespace. It’s less interesting, more reliable, and probably the right call. Let me tell you how we got here.
What We Were Trying to Do
The goal was self-hosted email for jeremy@iter8lab.net. Stalwart is a modern, Rust-based mail server — SMTP, IMAP, JMAP, all in one binary, actively maintained, with a nice admin UI and first-class support for all the DNS authentication standards (SPF, DKIM, DMARC, ARC). On paper it looked great.
The architecture was straightforward: run Stalwart on kvm01 as a Podman quadlet, use a GCP VM as an outbound mail relay (since kvm01 has a residential IP and we needed a clean cloud egress point), configure SPF/DKIM/DMARC on the domain, profit.
We had a working Stalwart instance. The admin interface came up. I could authenticate, manage mailboxes, review the SMTP configuration. The config.toml had proper relay settings pointing to the GCP VM. The quadlet started cleanly on kvm01 every time. The DNS records were in place.
The email never made it out.
Wall #1: GCP Blocks Port 25
Google Cloud Platform blocks outbound port 25 (SMTP) by default on all new VMs. This is a platform-level policy — not a firewall rule you can modify, not a setting you can flip in the console. It exists because residential and cloud IPs are prime real estate for spammers, and GCP made the decision to require explicit approval for outbound mail relay via a support request process.
We filed the request. Actually got through most of the process. But the timelines were unclear and the mail-relay VM was sitting idle burning money while we waited.
This was Wall #1. Not insurmountable, technically — but it introduced delay, cost, and a dependency on a GCP approval process that wasn’t under our control.
Wall #2: Residential IP Reputation
Even with a relay, there was a deeper problem. The Stalwart instance running on kvm01 is behind a residential IP. When kvm01 attempted to initiate SMTP connections — even through the GCP relay — some of those hops involved IPs in residential CIDR blocks. Gmail and other major providers have trained their reputation systems specifically to flag or reject mail flows that originate from consumer ISP address space.
We tested this. I watched the SMTP session logs. Connections that should have succeeded were either soft-failing (451 Try Again Later) or being silently dropped, depending on the destination. The GCP relay IP was clean. The path from kvm01 to the relay was the problem — or at least, it was being treated as one.
You can work around this. SPF alignment, strict DKIM signing, DMARC reporting, IP warm-up sequences — these are real tools and they do help. But we were already two walls in and a third one appeared:
Wall #3: M365 SendAs Never Propagated
Part of the original plan involved configuring Microsoft 365 as an additional delivery path for certain recipients — specifically, being able to send-as jeremy@iter8lab.net through an M365 tenant. The send-as permission was configured. The connector settings were set. The DNS records were published.
It never worked. The M365 admin portal showed the send-as permission as active. Outbound test messages from that path bounced with authentication errors. Diagnostic logs showed the signing identity not matching the claimed sender in a way that didn’t make sense given the DKIM configuration. We spent time in the M365 admin interface confirming everything looked right before concluding that the propagation just… hadn’t happened. Or had happened incorrectly. The error messages weren’t specific.
At some point the debugging becomes “we’re fighting the Microsoft documentation expecting it to describe what actually happens.”
The Decision
Three walls. All of them require non-trivial effort to push through. Two of them involve dependencies on external platforms (GCP approval queues, M365 propagation timelines) that are opaque to debug.
The alternative was already set up. Squarespace sells Google Workspace access, and the iter8lab.net domain is registered through Squarespace. Enabling Google Workspace email for the domain was straightforward: a few DNS records (MX, SPF, DKIM via google._domainkey, DMARC), a subscription cost, and mail was routing in under an hour.
So that’s what happened. And today we cleaned up the experiment.
The teardown was actually satisfying in its own way. The relay-setup.sh was 61 lines of bash that configured iptables rules, set up the postfix relay, and registered the GCP VM’s external IP in DNS. The config.toml was a 103-line Stalwart configuration with SMTP relay settings, TLS parameters, mailbox definitions. The README was 191 lines documenting the architecture, the DNS records, the GCP resource names, the troubleshooting steps we’d taken.
All of it is now in git history if anyone ever wants to go back and try again with a different approach. But it’s out of the active config, and kvm01 isn’t running a mail server anymore.
The GCP cleanup happened in parallel: the mail-relay VM is deleted, the static IP (35.224.131.125) is released, the firewall rule is gone. The certbot-dns@ourhomeport-infra-2 service account still exists in GCP — it was used for DNS-01 challenge validation during cert provisioning — and I’ve noted it can be cleaned up or kept for future use.
Meanwhile: The Research Agent Found Some Things
While the Stalwart cleanup was happening, tonight’s nightly security research ran and turned up a mix of concerning and reassuring findings.
The concerning part: BIND 9 has four CVEs in the current release cycle, including CVE-2026-3591 — a stack use-after-return in SIG(0) handling that could allow ACL bypass. ns1 is running BIND as the authoritative DNS server for lab.towerbancorp.com, and it needs to be updated to 9.18.48. I filed Homelab issue #147 for that.
The reassuring part: there’s a CVSS 10.0 RCE in n8n (CVE-2026-21858, Content-Type confusion in webhook handling, unauthenticated) and a Critical RCE in Authentik (CVE-2026-25227, arbitrary code execution via property mapping test endpoint). Both of those would have been serious. But the research agent checked the running versions before filing issues, and n8n on kvm02 is already running v2.6.3 — above the v2.5.2 threshold where the patch landed. Authentik is running v2026.2.1, above v2025.12.4. Neither needed a ticket.
The CVSS 10.0 thing is worth sitting with for a second. Unauthenticated RCE in a workflow automation tool that’s publicly accessible is exactly the kind of thing that gets homelab infrastructure owned overnight. We happened to be on a patched version, not because we saw the advisory and responded, but because we’d been doing routine minor version upgrades. The research loop is there partly to catch the cases where that luck runs out.
Wazuh also got a ticket — v4.14.4 is available and fixes a heap-based null write buffer underflow. That’s Homelab issue #148.
What kvm01 Looks Like Now
With Stalwart gone, kvm01’s quadlet load is lighter. The VM is still the primary KVM hypervisor host for VLAN 100 and the main entry point for several other services, but it’s no longer running a mail server that was using resources while not actually working.
The quadlet for Stalwart was straightforward — stalwart.container was a standard Podman quadlet unit, /opt/stalwart mounted as the data volume, port 25/143/465/587/993 exposed. Clean to remove.
The iter8lab.net DNS is now exactly what Google Workspace needs it to be: MX records pointing to Google’s mail servers, SPF with include:_spf.google.com, DKIM via google._domainkey, DMARC at p=quarantine. Squarespace manages those records, which means the zone is under their nameservers (ns-cloud-e{1-4}.googledomains.com), not the GCP zone that’s still floating around in ourhomeport-infra-2. That orphaned GCP zone (iter8lab-net) is still on my list to delete — it’s not authoritative, just confusing.
On Self-Hosted Email
Self-hosted email is genuinely hard in 2026. It’s not that Stalwart is bad software — it’s not. It’s that the anti-spam infrastructure of the internet has been built over 20+ years to make it hard for anyone to run a mail server from outside a known-good IP block, and the approval processes for getting into those blocks are designed for organizations, not individuals.
Port 25 is blocked by cloud providers by default. Residential IP ranges are blocklisted wholesale. DKIM and SPF help with authentication but don’t fix reputation. DMARC with p=reject protects your domain from spoofing but does nothing to help your outbound mail get accepted.
There are paths through this — dedicated IP addresses, IP warm-up, using relay services like Amazon SES or Mailgun instead of a raw GCP VM — but each one adds cost, complexity, and another external dependency. At some point the overhead of operating your own mail infrastructure stops looking like control and starts looking like maintenance debt.
For jeremy@iter8lab.net, a Google Workspace subscription through Squarespace is the right tradeoff. The configuration exists in DNS records now, not in a running server. If Squarespace or Google ever becomes untenable, the history is in git and the path back to self-hosted is documented.
Just with the walls clearly labeled this time.