Ask HN: The middle ground for email self-hosting? There were many threads previously on how bad/impractical it is to self-host your email. For example, this thread[1] is just a few days ago. I was wondering if anyone had any experience combining the microsoft 365 business basic (6$ a month) with self-hosted email server? By relaying SMTP through the Microsoft provided outlook server, would my custom domain be free from being marked by spam? |
For personal stuff I’m using iCloud+ domain hosting now after moving from fastmail which was also trouble free.
Setting up an email server is somewhat easy. Ensuring that other servers consider yours as legitimate that is the problem.
Obviously GMail is such a giant that email providers have to be very careful when blocking it, but enough spam comes from there that receivers clearly use some heuristics to block some of it. I’ve even received multiple rejection notices because the GMail server my email was sent through happened to be on a blacklist!
I switched last year to sending directly from my VPS. It was partly for privacy from Google, but moreso so I could enforce outgoing TLS. For the first few days they went to spam boxes or moderation queues, but I made sure they were rescued, and ever since I’ve had no deliverability issues sending to Google, some local ISPs, and even Microsoft (which seems crazy, as I never got a mail from my domain to show up in Outlook when I was relaying through Google).
I can only speak for my own experience, of course. But that is what I experienced.
The easiest to work with have been Microsoft and Yahoo. I still haven't found a way to whitelist our IP on centurylink.net, charter.net and att.com (please let me know if you have any ideas)
Some links that might be of help to others:
[1] https://sendersupport.olc.protection.outlook.com/snds/data.a...
[2] https://io.help.yahoo.com/contact/index?page=confirmation&lo...
[3] https://postmaster.google.com/u/0/dashboards#st=userReported...
[4] https://support.microsoft.com/en-us/supportrequestform/8ad56...
The result would be a screen or two of auditable gobbledygook that took a week or so to write. Somewhere in there, there'd be a pointer to an EC2 or ECS image, among other things.
It'd be a learning experience, and very tied to AWS. I wonder if someone else already did this.
Doing things perfectly and building up "reputation" helps a lot.
Nowadays, I just use mailcow (https://github.com/mailcow/mailcow-dockerized) for the setup part and have a much more polished experience. Email deliverability is not a problem. Generally, you just have to make sure to correctly setup DKIM/SPF (and DMARC) and check if your IP is on some blacklist. You can get it removed easily. (Edit: Also required is forward-confirmed reverse DNS, see below).
There was one provider that denied incoming mails from me, even though I got the IP removed from every blacklist I could find. I wrote a short mail to the admin contact and got told I had to host a web page with contact information on the same IP. Since being whitelisted there, everything works like a charm, couldn't be happier.
Interesting. Was that just to prove to that particular provider that you, the emailer, own the domain? Or is it some more widely used (beyond that provider) practice?
They require that
a) the sending IP address has a PTR record b) from a domain that you own c) that resolves to the same IP address.
This is also very important for general deliverability (https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS).
They furthermore recommend d) that your host name should clearly mark it as a mail server and e) to make sure the domain leads to a web page that contains provider details and contact information.
T-Online?
Incoming email to my domain is forwarded through Cloudflare's free service to the generic Gmail account.
This seems to pass all quality checks to avoid being sent to spam.
The only issue is that emails don't come in instantly. They can take up to 15 minutes. I consider this a feature, but it would be really annoying to some.
I have a somewhat similar setup, slightly different. Most emails arrive within 10-15 seconds, but sometimes gets stuck somewhere along the line (as it happens with emails), which is normally not a problem.
But some platforms force you to use the "send link to login via email" option for login, which again, normally is not a problem. But when they have a timeout of 10 minutes + it takes 15 minutes for it to arrive, you end up not being able to login.
Only happened to me a few times during the years of this setup, but when it does happen, it really sucks.
Seems like it shouldn't be hard to check and collect reference statistics with a survey, though I'm failing to find surveys of that kind, and getting accounts on public services would be the tricky part for me personally (since I don't like to provide my phone number), so not doing that myself either. Only occasionally tried to check it with others, and messages were delivered fine in those cases -- but that's just a few samples.
You might have a bit of SPF fiddling to do, just because you might be fighting the default self-hosting assumption that incoming and outgoing servers are the same.
> would my custom domain be free from being marked by spam?
The receipient's mail service gets to choose if it thinks your email is spam, this will happen whatever your sending arrangements, outlook is not immune from sending spam and is no magic guarantee others will give it a free pass somehow.
Recipients score your email on a variety of characteristics, many of which are under your control. A major consideration is the sending netblock, eg, residential ADSL blocks are likely to be rejected or scored to hell. Garbage netblocks like linode with a terrible reputation likewise. A clean (no history of spamming) IP in a clean (reputable) netblock will be scored higher. You can look up sender reputations here, which is the service the big email providers use.
So to send your own mail, you should rent a dedicated server on your own IP, you can do this for $30/mo or so. All you need to run there is postfix + SASL auth to forward your (and only your) emails.
Then you must configure DKIM etc correctly and check your emails are validly signed, DKIM requires being able to add TXT fields to your DNS.
It's very possible to do this yourself securely after a bit of a learning curve and have it require minimal ongoing maintenance.
But this is your outgoing email authorized by DKIM... an attacker can use it to take over most of your accounts via Forgot Password flow. I think it is a false economy to have that depend on a shared VM.
Had an issue with my self hosted email going to spam and these services solved it.
Bottom line: There's no "middle ground", any middle ground you cede is allowing a third party some kind of access. Hosting your own email has become expensive and time-consuming (although IMHO it's still extremely worthwhile, and I do it in spite of what a pain in the ass it is). Be prepared to spend at least $50/mo and at least 6 hours in setup and 1-2 hours a month debugging if you do it personally. Or you can find someone to help (see below). You need your own IP address. You need a dedicated box, not a VPS. And check the IP address in advance to make sure it's clean, and not blacklisted. Tell the datacenter you're going to be doing email and ask them if they're okay with that for a clean IP. Use https://mxtoolbox.com/blacklists.aspx to test the IP address they're offering you, or IPs in their range. Unlike some people are saying, you should never do this off a VPS if you have an interest in keeping the email secure and functioning for a long time.
My personal go-to would be dedicated hosting in the Netherlands, Switzerland, Isle of Man or Norway. Clean IPs, your own box, start with a clean server. But then you're talking $250/mo or so.
If you don't know how to set it up, there are people who can do it for you. You will need to essentially trust that person with access to all your correspondence, but if they do it properly, no one at the server farm[0] or elsewhere will have access to your correspondence... which puts you in the 0.01% of people on earth whose email isn't read by big tech companies.
[0] -who doesn't physically access the server: Look for ones in cages and ask who has physical access and why.
To me the happy middle ground is email on your own domain but using an existing provider such as G / MS or whoever. That way you've got control but don't need to worry about the pain.
It does require paying for but really on balance not much. If you're spending more than an hour a year maintaining your self hosted email (which you will, big time!) then your Google Workspace / O365 is paid for.
The situation I've found frustrating is about family email on same domain. I've gone in a huge loop that has ended up back with GWorkspace which is quite costly for 3-4 family users. But still - not even close to the horror of self hosting...
This worked well for me because it gave me the feeling of having more control and privacy and security over my email.
I switched away from that solution when I realized that in practice I have less ability to effectively provide security than the whole security and product teams of a major email provider.
They're a domain registrar that also supplies email and hosting and similar services.
I use a single M365 Business Basic account, as a conventional mailbox, for one of my domains. From within the Exchange Admin Center there's extensive control over mail flow -- domains to accept mail for, inbound and outbound connectors for routing mail between on-prem mail servers. Best as I can tell, literally ongoing any subscription that gets you an account with access to EAC ought be enough to route any or all of your email through EO in either direction.
https://www.microsoft.com/en-us/microsoft-365/exchange/compa...
https://www.microsoft.com/en-us/microsoft-365/exchange/excha...
https://docs.microsoft.com/en-us/exchange/standalone-eop/sta...
https://docs.microsoft.com/en-us/exchange/mail-flow-best-pra...
https://www.pxeger.com/2020-07-02-hybrid-cloud-email-with-am...
It is a bit overcomplicated, because I also set up SES to receive email, but I could run that instead with an ordinary Postfix server. It would be much simpler for outgoing only, I think
People don't know this, I mean no one knows this, but if you're running an EC2 instance with some allocated IPs, you can contact Amazon's customer service and ask them to unthrottle outbound mail on it. Typically they clamp down and stop connecting any SMTP deliveries off the EC2s if you do more than 10 emails/hr or something. But if you plan on keeping the IPs for awhile you can set up DKIM and SPF and all that, call Amazon and tell them you're sending and receiving legitimate business emails off that server. They may try to refer you to SES, but if you tell them you need to manage it on a private server for legitimate reasons, they have the ability to lift the block for you.
Do check your allocated IPs for blacklisting in advance, and obviously don't give them any cause for being blacklisted in the future.
You can self host mail alongside gmail/outlook on your own domain. More than one email service can run concurrently, without any problems.
That often overlooked fact allows you to quickly set up something like gmail on your domain, then use the trial period to see if you can self-host with any success. If you can, then you can shut down the trial, or move on to trial another paid service like 365 while you're still "trialing" your own host.
It really helped me make the transition.
Doesn't solve privacy, data ownership, nor google lock-in issue (but at least if I lost my gmail, I can move to a real email selfhost solution and keep my address). As my need is just to have custom domain address for the cool factor of it, this simple setup works flawlessly.
I work on a email client and a large majority of "this email is broken" is due to weird outlook behavior. Most recently it's TNEF attachments: https://en.wikipedia.org/wiki/Transport_Neutral_Encapsulatio...
Though, others do violate standards as well, but not like this.
We have had this setup for several years. It is not difficult to setup, emails are delivered reliably, and email delivery cost is negligible.
Some of our users use Outlook / Thunderbird / Apple Mail as a client, some use GMail as a client (check external mail / send as user) and some use Rainloop which I set up on the mail server.
Truth is I like Gmail but I think Google have dropped the ball with, "Let me point all my custom domains to a Gmail account. I would even pay you but I don't want Workspace".
For example, email notifications sent by Stripe are delivered over TLS'd connections. My bank does this too. If you are to proxy these, the relay will obviously be in the loop on all emails that aren't local to your mail server.
It's damn cheap too, like almost free for low volumes.
I think Amazon uses this for their workmail also and has become pretty strict at policing abuse.
I am only speaking for gmail though, so ymmv for hotmail et al which I haven't checked.
Helm is a personal, private email server that won't share your data.
The Verge
My favorite is mailbox.org
If I send an e-mail to a company's customer support, or to my senator, or I reply to a potential client, or I contact an open source mailing list and I don't receive a reply - do I know if my message made it to them or not?
I mean, it's plausible that JohnDoe@senate.gov just didn't deign to reply to my e-mail. But it's equally plausible there's some subtle misconfiguration - like an e-mail forwarder that breaks the SPF signature. It's not like I can sign up for a senate.gov e-mail address to test with.
Meanwhile, to paraphrase an old joke, when your senator rejects your e-mails you have a problem. When your senator rejects @gmail.com they have a problem.
I've had three issues. The first was delivering to outlook.com, but this was temporary and resolved relatively quickly: I simply contacted their support. At the time, they didn't bother to validate DKIM or DMARC according to their headers.
The second was a sender sending to us with a misconfigured SPF policy. I had quite strict rules that spf failure => user's junk folder that I had to relax, but I also had a discussion with the admins at the sending company to explain the issue.
The third was yahoo. For reasons known only to them, they decided that IPs they've never seen before will be blocked by returning an smtp deferral that is permanent, which is bad for legitimate mail servers because the email remains stuck in the mail queue forever. I ended up discussing this with their support also and after some discussion that block too was removed.
That's pretty much it. I receive dmarc reports now from many providers so I've an idea what percentage of our email is quarantined or rejected (none). I've been running email since 2011, for my own main email and a few others. I don't think deliverability is that much of an issue and I was able to resolve all the problems I've had in 10+ years of doing this by emailing support, explaining myself and asking to be unblocked. Usually this simply resulted in "OK but if you do bad things we will block you no guarantee of inbox delivery etc etc etc". That's fine. It seems that there is a large degree of per-account spam filtering as well at the big providers mapping to individual users' preferences.
Of course, if you don't set up SPF/DKIM/DMARC, or you have an IP with poor reputation (you can check the DNSBL) or worse a residential address, you will have trouble. I would generally look for a provider that has a relatively strict acceptable use policy, and in particular doesn't allow VPN endpoints to be run from their infra for your email, to reduce the chances your IP has a terrible reputation with the big providers. Also, join all the sender programmes, set reverse dns, don't let your users do things like send bulk email and that will reduce many of the problems.
I had to work around them for some datacenter mail relays. The only solution I found was to sum up the number of mail relays behind a SNAT and then apply rate limits for their domain to not exceed 6 concurrent connections total per SNAT. To your point and AFAIK they do not document this anywhere.
Having people whitelist you on google / yahoo / msn because you explicitly ask them to does have a wider effect, as far as I can tell, of keeping your emails in the clear for everyone else.
That is pretty much it. One factor is that once you are on a blacklist it can spread like wildfire and be much faf to get off them all again, so the risk is small but the hassle if it happens is high. Also if you send mail for numerous people there is going to be a much higher risk: every extra user/account/address is an extra hack target (do all your users have good, non-shared, passwords?) or just extra volume that might be accidentally classified as junk (and once something from your server gets classed that way, future content may get more aggressively analysed and more mistakes may happen).
I've run my own mail server, including sending mail directly, for many years and to my knowledge not had a significant delivery problem. But I have a few mitigating factors: the IPv4 address is essentially on a commercial ISP range, not one that looks like a residential account or a VPS service provider, and the ISP is one that takes junk mail seriously, so there is less “splash damage” potential, and the same range has been used this way for several years (the main sender has moved around that small range, when testing upgrades on a copy VM for instance, but never away from it entirely) so it never looks like a brand new mail server these days, I only serve myself and a very small number of other users, our outgoing mail volume is pretty low.
It is a bigger problem for hosting services (much bigger user-base and little control over what they might send) or if you are sending from one of their ranges, if sending from a residential ISP address range, if your volume is high (perhaps you have apps that send mail as well as your personal mail?), etc., but it can be a problem for everyone.
I'm rebuilding my mail service soon (moving off Zimbra to just configuring the parts myself, as we don't need the extra features these days, it is too chunky for just a mail server, and at the end of next year they stop releasing easy install packages for the non-paid users (they already have for v9., next year v8. hits EOL)) at which point I might reconsider where it is hosted and if I should be sending via a paid SMTP relay to let them worry about deliverability, though as far as I know I've not had a problem.
It's not just misconfigured email server settings like DKIM, SPF, DMARC etc. One can correctly set all of those and still have the outgoing emails rejected or spamholed. Why? Because the big email players like GMail, Microsoft Outlook.com, etc use black-box heuristics of reputation datapoints that exist outside the boundaries of email settings such as... "amount of email volume", "# of spam abuse reports from ip block", etc.
Because "sender reputation" cannot be encoded into an email configuration (DKIM/SPF/DMARC/etc), that's why nobody can provide a convenient Docker container with a perfectly working self-hosted email server that can reliably send email. If such a thing existed, the spammers would use it as well!
A datapoint such as "volume of email from this ip" is an unstated behavior/activity number and not an identity setting like DKIM.
And the invisible heuristics keep changing which causes previous email setups that worked -- to later stop working for no obvious reason. Why? Because there's a constant arms race between spammers and email filter algorithms. This means others' email spam heuristics that keep evolving and that you don't control -- blocks your self-hosted outbound emails without warning.
That's why you have example of skilled admins who know what they're doing and had a working self-hosted setup for years suddenly getting their emails rejected: https://www.tablix.org/~avian/blog/archives/2019/04/google_i...
As to the contradicting anecdotes about the difficulties of self-hosting email, the issue is that the conversation shares the same unstated environments in comments about Uber or umbrellas that affects how the writer perceives the truth or relevance of their anecdote.
- "The problems of self-hosted email getting blocked is overstated. I've been doing it and it's working fine."
- "I'm not sure what value Uber provides. Taxi services have smartphone apps."
- "I'm not sure why people use umbrellas. Every time I walk outside, it's not raining."
As an example of evangelists and advice-givers not noticing their unstated environments... Back in October 2017, a commenter (lucb1e) argued[1] that I was exaggerating the difficulties of reliably sending email but a year later in 2019, he eventually confirmed the same difficulties! [2]
I've had this discussion on HN before. It's gotten to the point where I've had to have my clients and their corporate lawyers go to bat against mail providers to maintain deliverability. No mail provider has any interest whatsoever in allowing an independent mailserver to continue delivering now.
So far, legal threats have worked when push came to shove against certain networks. But I imagine the difficulty is only going to increase.
That’s simply not true in that generality. I run my mail server on a VPS costing ~$7 per month (have been for ~20 years, switching the VPS provider once in that time) and mostly only have to do something when I major-upgrade Debian every 2-3 years. (Security updates are automated.) Some of my friends do the same. For the initial setup I would plan for more than 6 hours, it can certainly take some effort to work through all the details.
I agree that hosting a mail server directly on a VPS compromises privacy and control. But there’s a better alternative: use VPSes for cheap static IPs, while hosting the server locally on hardware you physically control, using WireGuard tunnels and port forwarding to connect things. Port forward incoming SMTP over WireGuard to your real MX, and use MTA‐STS and DANE so that as many senders as possible will TLS‐encrypt mail they send you. Have your outgoing SMTP server handle DKIM signing, then send it out via WireGuard so it looks like it came from the VPS, while enforcing TLS encryption.
The VPS won’t be able to forge mail from you without your DKIM keys. It won’t be able to read your outgoing mail due to TLS. It won’t be able to read incoming mail that’s TLS encrypted. It will be able to read unencrypted mail, but the big providers that follow MTA‐STS will abort if the VPS attempts to block encrypted connections.
This has the added benefit of reducing your dependence on an external provider (the VPS company) for server setup. If you’re unhappy with a particular provider, just switch to another one. The issues associated with sending email from a brand new IP will be there, but you won’t have to set up complicated infrastructure on the new host, only a few WireGuard tunnels and firewall rules.
If you're paying for that, why not just pay for a static IP at home?
> You need a dedicated box
> But then you're talking $250/mo
Nonsense.
You only need a $2-5/m cheap VPS for your server and have anything as MTA/smarthost for you. Eg: Fastmail, $5/m, just receive with a catch-all address, send through a SMTP with auth (with an app pass).
No bother with SPF, DKIM, PTR, SPAM and all this bullshit. Also you have a breaking glass access to your mail in case your server burst in flames.
Don't want Fastmail? Any other provider would do.
Want to tinker it yourself? Just buy anything anywhere, configure it as a smarthost.
Anecdata: I have a small business of ~15 people use Fastmail in the said configuration. The only difference is what they are on Exchange Server locally. The old one, on premises. They use it for almost 8 years without problems.
It has a few advantages because I can run a bunch of automations and filters. Honestly I get less spam than in my gmail inbox, so I harvest spam from gmail to train my filters now. It's insane to me how much providers charge for an inbox and the tiny sizes many still offer
But what's the cost when Google's "AI" bans your account?
For paid email hosting I'd go to some provider with actual support...
For several years I’ve hosted in the “middle ground” sense described by the OP, running my own incoming mail server and relaying outgoing mail through a big provider.
The main benefit for me (compared to using a big provider with my own domain) is personal privacy. When I used Google for mail, Google had access to so many pieces that make up my personal life: Purchase receipts. Flight itineraries. Conference registrations. Emails from my university. Emails from my realtor. Utility bills. Notifications for subscribed forum threads, GitHub repositories, Wikipedia pages. Whatever newsletters I chose to subscribe to. Theoretical access to any site with password reset by email. Running my own MX eliminates Google’s access to most of these things.
There are other some other benefits too. Free infinite aliases I can use to sign up on any website. No fear of dependence on features that might get paywalled. No sudden danger of having to migrate data to another provider.
> If you're spending more than an hour a year maintaining your self hosted email (which you will, big time!) then your Google Workspace / O365 is paid for.
Reducing my data footprint is something I care about enough to spend my spare time on.
My basic strategy is one of slight defeatism, I have to admit. I am 100% in to Google for their (really quite excellent) tools in Google Workspace: nothing is as good as GDocs, nothing is as good as Gmail, nothing is as good as Google Meet; but I do things to ensure I'm not utterly f**d if the Random Google AI Best happens to decide I'm some sort of unspecified menace. So for instance - I use Google Docs but only with .docx / .xlsx files rather than native .gdoc .gsheet files. I back this up automatically to my self-hosted NAS. I do this on a domain which I own, so can step away if things do happen to go south, or costs double or whatever.
Then I use kagi.com for search, and have a piHole / ublock / Brave to minimise footprint from a tracking POV.
I know, it's all probably moot given I just open up my inbox to Google, but I've tried and failed to find a provider that is even close to being the same balance of low price + utility. I got excited about Fastmail but turned out it was a combination of not very good AND really expensive once I factored in having several accounts on the same domain. I had a horrific experience with iCloud+ (they have a weird YEAR long account blocking issue thing that I won't go into now). M$ was awful and required me to send everything through GoDaddy's DNS. All the others were just underwhelming or expensive or both. So - sadly - I'm back in the G stable where I'll stay for the time being... :-)
What this does not do is trigger "undeliverable mail returned to sender" messages, so the end user has no idea their message is stuck in their own MTA's mailqueue until the MTA decides it has tried enough and gives up, and MTAs will usually persist for quite some time.
Spammers won't care what error code you send them or even worry about deferral messages, which is why the temporarily deferred spam trick works (first send is deferred for 1 hour, if you are a genuine MTA and you try again respecting this, it works). But permanent deferral, as I say, is very user hostile. The user thinks they've sent an email, but it isn't in the spam folder of the recipient. The sysadmin then has to go and dig to find out what exactly happened, and remove the mail from the mailqueue.
Luckily so far as I know we are only emailing a single yahoo address.
tl;dr the technique they are using is designed to handle the case where the receiving MTA is offline temporarily. There is a spam defence trick you can use and I don't object too much to that, but they used it to implement their block list rather than outright rejecting, and set the timeout to deferred indefinitely, which is just bad.
Yes, that and RDNS.
> If you're paying for that, why not just pay for a static IP at home?
That’s a good question. I too hear that mail providers consider IP blocks assigned to VPS providers less trustworthy than others. The reasons I don’t take the ISP/dedicated server route, aside from price, are:
• VPS providers are not tied to my physical location. If I move, I probably can’t take my ISP’s static IP with me (I may even move to somewhere they don’t service). Conversely, if I want to switch away from a local ISP, the selection of alternatives is extremely limited.
• Risk of neighboring IPs reducing the reputation of the block exists with server companies and local ISPs as well. I concede that the problem is probably worse with VPSes, but I hope to mitigate it somewhat by avoiding bottom‐of‐the‐barrel providers and by the fact that my own IP will never be used to spam.
• I’m somewhat worried about the possibility of DDOS, and VPS companies provide a lot of cheap bandwidth, so in case of attack I might be able to salvage the situation with careful firewalling on the VPS.
>> I’m somewhat worried about the possibility of DDOS
The one time I got severely dDoS'd, because I'd let a friend run a tiny static website off my server that attracted that kind of attention, the hosting company I was with shut my account down immediately and asked for $5000 in reparations. I had to backdoor into the server and salvage whatever I could. That was a hardened box in a military grade facility. I don't think a VPS is going to be kind. Push comes to shove, if it's in your house you can pull the cable.
Now I'm terrified that I'll return one too many pairs of socks or something, and get the retail consumer part of my account banned (we buy a lot of stuff on Amazon).
Does anyone have experience with what happens to AWS resources (specifically, domains) when that happens?
That's not really my worry, though. What scares me about being heavily invested in blah@gmail.com or blah@notmydomain.com is lockout from services. As long as I have my own domain then I can just switch out the DNS and I'll still be able to get into whatever web service I signed up to using my email address.
If you don't control the server, you don't have independent email.
In the end, the UX of having to wait for an email sucks, sometimes.