I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.
Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc
This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
I use the form of hostname.int.example.com for everything inside my home network. None of which is accessible to the outside world. I use LetsEncrypt with DNS validation to get the certificates.
I'd prefer this over split DNS, any day.
Removing attack surface is better than trying to hide it.
Or even a more extreme example: https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain:
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net").
There are DNS servers written strictly focused on this use case:
* https://github.com/acme-dns/acme-dns
Also code that handles a bunch of DNS provider APIs so you don't have to roll your own for ACME client hooks:
https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.
Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.
I bet you can do it with HA-Proxy, but I use https://github.com/ThomasHabets/sni-router
Most browsers support trust on first use for leaf certs
Using a browser in an air gapped environment is so much more pain than it should be.
And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today
You now also have to build infrastructure to distribute the wildcard from (presumably) central place where you generate it to all the different places where it is desired.
And hope the wildcard's private key does not leak from one of myriad of places it now lives.
Leaking is an issue but we're talking about internal services too.
* "internal services" = on a single server that is publicly routable
1. Register a domain ("server.com") and put it on some public DNS that can do DNS validation with acme.sh.
2. Use DNS validation to get a certificate on your domain from Let's Encrypt. You can just grab a wildcard one ("*.server.com").
3. CNAME all of your services on a public DNS to an internal address ("email.server.com" → "server.internal", "plex.server.com" → "server.internal").
4. Resolve your internal address on a local DNS server with an A record ("server.internal" → 192.168.0.123). This can often just be done on your router.
Unless you enjoy that sort of thing.