there exist several pieces of folk wisdom:
- "you cannot run your own mail server in 2025, this is too hard and time consuming" (completely false, i've done this since ~2010 with minimal ongoing maintenance)
- "you can do it but gmail will sort your mail to spam" (partially true and what i want to talk about here)

recently, my hand was forced: i had to migrate my mail server across providers and regions. it's unimportant why but important what the result is.

1/3

in the end, i went:
- from running mail.whitequark.org on DigitalOcean (extremely poor reputation of all IP blocks) to Hetzner Cloud (coin flip, but you can get clean IPs with minimal effort)
- from running roundcube+postfix+dovecot on ansible+Debian with spamassassin (ongoing suffering) to NixOS with rspamd (one-time suffering)

in the end i have clean IP reptuation, SPF pass, DKIM pass, DMARC pass. does gmail accept my email? yup, cold emailing people just works. with everyone but M365.

2/3

by setting the entire thing (https://codeberg.org/whitequark/mail.whitequark.org) up from scratch on a never-seen-before IP using a completely fresh server in about a day on new-to-me NixOS, i want to put the folk wisdom of "you can't self-host email in 2025 and even if you did gmail won't accept it" to rest. evidently, it can be done.

Microsoft 365 however apparently will hate your email no matter what. you learn to live with it

3/3

_should_ you self-host your email? i dunno, i'm not your mom, pick your own poison. i do it because i need independence: throughout most of my life i have been, for various reasons, under significant risk of losing control of corporate hosted email in one way or another. doing it myself solves that.

_can_ you self-host your email? Absolutely, in under 500 lines of Nix code

@whitequark yep, I echo your experience here. I think (no evidence) that for some servers it helps if your server is well-established. but with most of them having DKIM and SPF is enough to make stuff deliverable

ACAB includes Microsoft.

@domi pretty much!

@whitequark

I wonder how much this worked because you had a clean reputation for the domain. I’ve been using the same domain for my primary email for 20+ years, so by now the big providers have it on their ‘probably not a spammer’ list. When I added some new domains. They used the same DKIM key and same sender IP, which probably helped. I didn’t have any problems with sending to Google or MS domains (when I was at MS, I tried sending a mail from my mail server pretending to be from a microsoft.com email address and it arrived too!).

But I’ve heard people setting up a new mail server have a much worse experience.

@david_chisnall this is possible! I haven't done a proper study of the factors that go into it so I can't claim otherwise. but, anecdotally:
- until I cleaned up my IP reputation (which has been awful for almost a decade) Gmail refused to deliver to anything but spam
- I always had clean SPF but never bothered with DKIM or DMARC
- I kept sending email under these conditions for years

@david_chisnall given this, what kind of reputation would I have? clearly not good enough to mark my email as ham; which makes it hard to believe it was somehow enough to result in a near-180 turn in deliverability after just a day? I'm skeptical

Confirmed: Microsoft won’t even deliver email you want to receive!

@whitequark

@futuresprog they do actually deliver it after someone from the org marks it as non-spam (and, especially, replies to it)

@whitequark Awesome, I have wanted to do that since forever. Unfortunately I don't have that much devops knowledge and always read that my emails would always go to spam, so in the end never tried. Maybe this is what I needed to read to finally give it a try.

@doragasu I do not have professional ops background either, I just dabble really

i think the myth of "you can't self-host email" persists because while it can evidently be done, basically all of the software involved is ancient, baroque, inconsistently documented, requires a PhD in Bullshit to correctly configure, and is almost actively hostile to observation. but this is _annoying_, very different from _impossible_, and fortunately mostly solvable by delegating the annoying bits to an expert using something like NixOS to make it reliably work

365 tenants are keeping their own known domains list?! Well that’s an interesting insight.

Public Outlook/Hotmail would send even well formed, with DKIM and SPF, and solicited email to Spam all the time

@whitequark

@futuresprog yep I believe so, assuming I understood your terminology right

@whitequark another reason that it persists is that even the public IP reputation scores are inscrutable as hell, and the secret bonus reputation scores that Microsoft, Google, Apple, and Yahoo all keep internally are even worse. As you are apparently discovering with Microsoft right now (sorry).

@glyph yeah I think the combo of "roll the IP gacha a few times" + "let it sit for 8 months while the VM idles" probably did me a lot of good here, though I can't prove it!

@whitequark yeah, the spooky magic parts suck.

HOWEVER.

while I would not encourage most people to do this, I also wouldn't want to unnecessarily scaremonger about the process, so there is a very important coda to this:

many people see the undocumented complexity, want to "keep things simple", and thus avoid DKIM, SPF, DMARC, and SMTP/TLS.

The key is to set up all the complicated shit on day 1. You can't control IP reputation but you CAN control these "security" markers, and they all help

something interesting I discovered is that the combination of
- greylisting (telling senders that are suspicious but not obviously spammers to retry in a few minutes)
- rejecting non-well-formed HELO hostnames
- rejecting clients that pipeline message data without even listening to your greeting
has reduced the amount of spam i receive (at the MTA) to essentially zero. i got one single message since i set this up. normally i get ~50 per day!

@whitequark
Unfortunately all of the spam mail I get is sent by the Big Email servers which properly implement this :(

these functions are now either built-in to postfix (smtpd_helo_restrictions, smtpd_data_restrictions) or come built-in to rspamd, so are very cheap to set up; I'm amazed that I receive all legitimate mail (personal and transactional) instantly but junk just stops coming. I almost felt like setting up DNSBL and Bayesian filtering was a waste of time

@whitequark the more you have this stuff consistently running clean (and don't get out over your skis with overly harsh SPF policies, especially if you ever forward mail anywhere; just *have* a policy, don't worry about making it strict) the more you are likely to be considered clean early on in the process.

(Ironically, the other thing that you want to do is… maintain a certain level of sender volume. So ideally do this for you and 100 friends not just yourself.)

@glyph also (a note to past catherine), DKIM doesn't involve regular key rotation so setting it up is way easier than say Let's Encrypt; you run a single CLI command, update DNS and kinda forget about it forever. or until your key gets compromised at least lol

@whitequark Tagging this as #ryoms.

@whitequark
Just like many others already mentioned: don't waste your time with Microsoft and emails. You will most likely end up getting insane or with a major headache at least.

@ben I managed to divine out just enough to know who and when to ping via other channels to get mail through; I don't think I'll ever get it to go through when it's unsolicited

@charlotte right yes, by no means is spam filtering obsolete, I was just shocked I got off this easily (and had to go test my configuration five different ways to make sure I haven't broken it)

@whitequark about "ancient baroque undocumented" software. Have you tried opensmtpd?
I kind of like it, it's more modern than postfix definitely.
But you have then to learn yet another config if you wanna migrate to it.

@yannsionneau I was migrating an existing server and wanted to keep it roughly 1:1

@charlotte but also if you haven't tried rspamd you might like it, it's much nicer to use than spamassassin

@whitequark
Yeah I have but I eventually just moved to a hosted mail solution

@charlotte oh rip

@whitequark
I forgot what the exact reasoning was but my setup was decaying a bit and incoming mail was hit or miss for some kinda important mail senders like my bank. Oh and ip reputation issues with one rbl that at the time I tried to get unblocked for the first time advertised me a paid but defunct "Email registration" program. Less spam though than with zoho

@charlotte yeah that all makes sense; i've always had impeccable reception (only spotty delivery), but my setup was decaying since the moment it was born and until i migrated it to nixos, so that was a major boulder off my shoulders

@whitequark
Oh it was decaying on nixos, couldn't be arsed to make it work better either

My best guess is that my last bank ran Email servers that haven't been updated for a decade+

also Roundcube is pretty amazing, they have a nice responsive new theme that works well on desktop and mobile and is as nice to use as the Thunderbird Android app if not more

one of the few PHP codebases i'd run in production

@simon_w i want to emphasize that i don't greylist everything; "clearly ham" or "50/50" goes through, "probably spam but not certain enough to reject outright" gets greylisted

works very well

@simon_w this means you can't greylist in the MTA and have to do it in the mail filter, but in exchange you don't get the annoying / flow-breaking delay

@whitequark you'd miss out on my email though, depending on your definition of "non-well-formed HELO messages'.

@bert_hubert try me? whitequark@whitequark.org

@yavien prestashop is ok. nextcloud i thought was ok but that was years ago and i'm less sure now

@whitequark as both a free outlook email user and an (rarely, but sometimes) organization m365 email user, i can confirm that i just blame microsoft for throwing anything unfamiliar into spam and learn to live with it,

the september discord *tos update* email was sent to my spam folder. i'm pretty sure there are anecdotes out there of microsoft sending their own notices into spam too.

@whitequark I recently went through a similar change. My old mail server was getting long in the tooth and, rather than upgrade the distro and potentially break all the configs in subtle ways I've forgotten, I searched for mail servers written in Rust and switched to Stalwart. It's been ... fine? ¯\_(ツ)_/¯ They make setting up all the things fairly easy (spf, dkim, dmarc, tls) with decent docs so I'm hoping future version upgrades will also be simple and painless.

@whitequark @david_chisnall > - I always had clean SPF but never bothered with DKIM or DMARC

This is something I see really often, and I think nowadays DKIM and DMARC are more of a requirement.

I see some organizations implement DMARC reject policies with SPF as their only mechanism. I don't think people realize how many mail forwarding services are out there that will always break SPF, DKIM is really crucial.

@whitequark @charlotte if you want to make it super easy, just use Stalwart which gives you everything in one. It handles your DKIM, TLS, spam filtering, SMTP/IMAP/POP, CalDAV, CardDAV, yadda yadda in a single Rust app.

@codimp which form? can i have a link?

@whitequark done

@bert_hubert received!

@whitequark For the second largest email provider, Microsoft appears to be deeply incompetent at handling mail. From my experience Outlook autoresponders from 365 accounts *always* fail DMARC and most annoyingly, Microsoft doesn't handle error messages correctly: Your server sends a standard "message size exceeded" and Microsoft tells the sender it was rejected as spam.

@bert_hubert @whitequark Just curious - do you have a reason to keep your HELO / EHLO name, as you put it, "non-well-formed"?

I wrote my own sendmail rules for checking HELO / EHLO more than twenty years ago, and I have yet to find a legitimate example of what I'd consider "non-well-formed" that I'd want to allow, so I'm curious about how you have yours and if you have a specific reason for it.

I have two kinds of checks. The basic check just makes sure the HELO / EHLO is a real hostname that resolves in DNS to real A and/or AAAA record(s). The other check makes sure the given HELO / EHLO is a real hostname that resolves back via A and/or AAAA to the connecting server's IP.

Would yours get caught in zero, one or both of these?

@AnachronistJohn @whitequark I send out 100k legit email messages/month or so, an they arrive at all major email players. I do not control the reverse DNS of my IP address, and other people who make rules force me to have my reverse and my HELO hostname to match.

@bert_hubert @whitequark I never thought that there should be a requirement for HELO / EHLO to match reverse DNS, although it's nice and should be done, if possible.

OTOH, one of my checks on my non-primary MX is to block all email coming from IP addresses that have reverse DNS PTRs that either don't resolve or that don't resolve back to the corresponding IP.

I'm much more ready to block on non-primary MX because spammers still try to deliver to them even when there's no issue connecting to the primary MX.

So your HELO / EHLO isn't what you'd prefer, but I take it at least it resolves back to the IP address the connection is from?

@AnachronistJohn @bert_hubert hah, looks like my primary rejected Bert's email but the secondary delivered it after all; I never thought of this before so thank you for the insight!

@whitequark oh hell yes these 3 SMTP tricks definitely cut out about 90% of spam. One more I do is to wait 5 seconds between SMTP client commands and server responses including the initial HELO because that will catch even more pipeliners while not affecting delivery of legitimate email at all.

@jpm @whitequark Right, I've got this implemented, too.

permit_mynetworks, sleep 2, permit_sasl_authenticated, sleep 2, reject_invalid_helo_hostname, check_policy_service inet:127.0.0.1:10031, permit

@jernej__s @jpm oh that's super clever

@whitequark Let rspamd handle those checks, because it can't learn from mails it never gets to see if Postfix rejects them before they even arrive at the milter interface.

@wrmsr good point; do you have a configuration handy?

@whitequark @me_ I have been running my own for many years, as well, and keep pushing on this.

i’m curious, was your approach with SPF, DKIM, &c to iteratively add things, or did you just figure you should cover all the bases? I found SPF to make a big difference for deliverability, but haven’t needed the rest. I have found *1* correspondent (hosted on M365) which simply will not *send* me email because I don’t have a TLS cert.

Oh and att.net specifically hates my IP. Whatever.

@a @me_ I ran email with SPF only for years and had issues (maybe due to no DKIM, maybe due to IP reputation, or both); with the new server I decided to get 10/10 on mail-tester.com

@whitequark @me_ Also, it is infuriating when people use “can you get past Gmail’s spam filters” as a metric. *Google* can’t get past Gmail‘s spam filters. A couple years ago I had a Google workspace account hosting a Gmail list and email sent to Gmail accounts almost uniformly ended up in the spam folder, even though these people had explicitly opted in using Google‘s tools.

@whitequark seems like a good opportunity for a docker image with a solid config readme.

@twipped yup! I run too much infra to have capacity for this but other people can step in!

@whitequark there are IPs that have never been used before?

@twipped I'm exaggerating a bit; I sat on the IP for 8 months before setting up the server (for reason of being tired)

@whitequark could you explain what you mean with "cleaned up my IP reputation"? I'm really curious how that could be done.

@david_chisnall

@tyrthecat @david_chisnall I switched IPs because my old one has been perma-blocklisted by one of the big DNSBLs (due to DigitalOcean being bad)

otherwise you just fill out a bunch of forms

@whitequark thank you for sharing this thread!
It is super informative and has so many good tips to dig into details and better understand things

@tyrthecat you're welcome!

@whitequark 💞greylisting. It cuts down on so much spam it's like witchcraft.

A side effect that drives me nuts tho is 2FA via email.

It seems that 2/5 minutes tokens are becoming more common-place with each passing day. These stupid a*holes treat email as if it was instant messaging.

Sometimes I need to request 3 tokens to get a valid one to go through grey in time

My brother in Christ you're not a well known service and your tokens don't even come from your own domain. Chill the f down.

@HeyQui I greylist only email that is suspicious in first place (via rspamd); tokens and stuff go right thru

@whitequark another provider that will hate you no matter what is T-Online/Telekom which is somewhat common in .de.
They will flat out reject anything from an unknown domain and you have to contact them and show a working website on the domain with an „Impressum“ containing a physical address and phone number. I think that might even happen when you bring your custom domain to e.g. Fastmail or any other big email provider with good reputation.

@sven haha I have stalkers, guess t-online just never gets any of my mail then

@zimzat let's test deliverability?

i'm seeing that a lot of people know about greylisting but have various issues stemming from the added delay

my setup doesn't have this problem! rather than greylist at the MTA (Postfix), i greylist at the filter (rspamd); only email that is suspicious in first place gets greylisted, while email that's "known good" or "50/50" goes through

this doesn't noticeably reduce the efficiency of greylisting, but makes the mail server wayy more usable, especially with "magic login links"

@whitequark I've been running pretty much that exact setup for over a decade, and moving it to nix like that for a declarative config sure looks appealing. I've only played with nix a little, so reading a more complex config was educational, thanks. I'm definitely stealing that "load the config variables from a TOML file" bit next time I touch it.

@defer I don't really like the Nix language (while fully appreciating its sheer power!) so I design my systems in a way where merely operating them doesn't need editing Nix files. it also makes for a very nice conceptual split

@jernej__s @jpm which section do you put these commands in?

@whitequark

There are several shrink wrapped FOSS mail server packages around, which will do all the heavy lifting.

I use Mail-in-a-box for my mail server, and I spend less than an hour a week keeping in going.

@seindal a hour a week seems _incredibly high_

@whitequark

Less than ...

Also, I use it for work.

@codimp ohh I see, I haven't ever got rejects, only filtering into spam

I have all of this and my spam detection rate is way lower than yours. HELO catches some of my spam, pipelining catches nothing these days because all of my spammers seem to know they're not supposed to pipeline. For greylisting I'm not sure how effective it is.

Looks like I have to implement additional anti-spam defences like DNSBL or SPF.

@whitequark

@hallunke23 highly recommend greylisting

@whitequark I used to do it until the raid controller in the machine that was hosting it on shat a brick (Technical term) and it took me several days to get the provider to replace the faulty hardware (It was a dedicated server) and get everything restored from backups. Particularly as I couldn’t focus on it full time due to needing to do the dayjob

it was at that point I decided it wasn’t worth it for me.

@Dragon this is exactly why I use NixOS, which lets me restore a system from a backup in around 10 minutes

@whitequark The problem was more the provider being slow replacing the server.

Thesedays I’d just spin up another VM, this was quite a few years ago where “cloud” was less common.

Also for personal stuff operating at the budget end of the market didn’t help.

@Dragon right, yeah

i used to be way more conservative about putting everything on one single machine until the monthly cost of running a server didn't get comparable to monthly cost of renting an IPv4