EMudhra Intermediate CA Compromise(emudhra.we-have-a-problem.com) |
EMudhra Intermediate CA Compromise(emudhra.we-have-a-problem.com) |
https://www.nseindia.com/get-quote/equity/EMUDHRA/eMudhra-Li...
D=4. That's the entire private key for one of the certs.
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
IOT devices famously never get firmware updates, which means most of them will remain vulnerable to this leaked trust chain until the cert expires in 2039. That means you can forge an HTTPS server for any domain that an IOT device connects to and act as if you are the authoritative server. This means intercepting encrypted traffic and being able to decrypt it, or serving payloads to the device from a fake trusted server.
If your TV connects to whatever.lg.com, you can create a certificate for that domain using this intermediate signing cert, and pretend to be that server. That's how https://dv-g3b-emudhra.we-have-a-problem.com/ was created.
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
This means that the people with these keys could MITM almost any https connection people make, not only the ones that this CA issued.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
SO, not all CA's are signed by root certificates already on your machine
This has always been incredibly stupid.
Browsers might've gotten their magic revocation updates, but a lot of other things haven't… (and yeah this is "security by hoping hard enough", but still better than nothing.)
https://emudhra.we-have-a-problem.com/ tests all certs. If you don't trust that for whatever reason, https://bgp.tools/ also shows a banner at the top if you're vulnerable.
(from this author)
There is reason why company shouldn't cheap out and use foreign vendors like this....
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
You say this like it's easy and simple and we are simply not choosing to do so.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
Well, yeah. DANE should have won. NameConstraints would be nice tooo, but who's going to make CAs use it?
And FFS, clients could at least check CRLs once in a while.
Basically, the design didn't have a way to distrust anything other than removal from the root store. Which may still happen, but that will take longer.
Is the update cadence of CAs that different between the sources? Is the list of CAs included so different from OS vendors and the package managers?
Can you elaborate on this innovation in the email space around PKI?
Is it a out MTA-STS or something else?
In general, email has used DNS to a greater extent than the web has to build trust mechanisms that don't depend on giant lists managed by a tiny number of browser vendors.
Well you might think, oh I can monitor the certificate transparency logs.
I would say well, good luck downloading and parsing terabytes of certificates as they are logged. Then there are CAs whose logs will rate limit you in a way such that you can't catch up to the certs being logged with one server and IP. The system might "work" but not for domain owners. Good luck.
From my limited experience (using them at home) they seem to be fairly well supported in client libraries, actually!
- OpenSSL has full support, https://docs.openssl.org/master/man5/x509v3_config/
- GnuTLS has full support, https://www.gnutls.org/manual/html_node/X509-certificate-API...
- go (crypto/x509) supports everything except DirectoryName constraints, https://pkg.go.dev/crypto/x509#Certificate.Verify
- Python (pyca/cryptography) has full support, https://cryptography.io/en/stable/x509/reference/#cryptograp...
- Windows/Schannel has full support, https://learn.microsoft.com/en-us/windows/win32/seccertenrol...
- Java has full support, https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
- I know that NSS also supports them from experience but (sorry!) couldn't find the docs in a few seconds of searching
- I know that Strongswan supports them well because I actually opened an issue about it a few years back and got it fixed :-)
(Just a few off the top of my head)
…but, yeah, the issue is on layer 9. Even just nailing sub-CAs to ccTLDs breaks the moment a marketing person registers a .lol domain.
Then again, the CA/B is essentially a layer 9 entity. I just refuse to believe this is an unsolvable problem (though it might need NCv2 with a whole bunch more shit to cover more human creativity in policymaking.)
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
To be fair, unless I'm misunderstanding, wouldn't this be a good move?
LE has a good security record and takes itself seriously, versus these smaller CAs that seem to mess up regularly.
All things turn and curve and degrade; Google used to be the "it's perfect" darling.
> And what happens when you want to switch?
You have to update your pins and then wait long enough that you don't mind losing users who never update their software. When it's just in software you distribute, you can change hostnames in new versions and use the new CA for new versions and the old CA for old versions...
For web browsers, there's not a reasonable way to make a nicer transition. You have to serve old certs for old pins, and TLS handshakes from browsers don't include enough information to know if the client has an old pin or a new pin; or at least, mostly they don't --- if you get your pins changed and the browser TLS stack changes so the handshake changes after that, maybe you can use that... when sha1 certs were phased out, it was somewhat popular on high volume sites to look at the TLS Client Hello to decide to serve a sha256 cert or a sha1 cert.
DNS CAA only pins issuance, not acceptance; clients can not check if the CA that issued a cert was listed in DNS CAA at the time of issuance, and they do not check if it's listed at time of presentation. I'm pretty sure issuing a cert contrary to a CAA record is a mississuance and would result in sanctions from the CA/Browser forum, but clients would still accept the cert until it's revoked.
There was briefly browser support for site initiated pin on first user (HTTP Public Key Pinning), but it got dropped as mostly no one was using it and it was easy to get wrong and break your site.
The broader issue is that users don't want to (and probably don't have the ability to) actually make any trust decisions; they much prefer to delegate all of the decisions on trustworthiness to somebody else. The PKI root CA system does have its flaws, but on the whole, I suspect that people like Mozilla are better stewards as to whether or not the person vouching for somebody else is trustworthy than the average or even the above-average user.
I'm not sure I'm familiar with what moment this is referring to; can you elaborate?
To your second point, that might be true in a world where the consequences of making a bad trust decision was some kind of hassle related to changing passwords or reporting your cards stolen. But we seem to be moving towards a world where it lands you in a detention facility or some other dire consequence. I think that sooner or later, users are going to realize that they want that control and will be willing to do the corresponding trust hygiene work. Besides, isn't making that kind of work easy the point of technology? We just need to do better re: making it usable rather than investing in systems we know to be unsafe.
I'm not sure that's really true. Human social organization has, until around 10,000 years ago, been centered on small groups of people where everyone literally knew each other. The ethnographic studies on non-state complex societies generally tends to show a lot of kinship-based networks that developed, and even in state societies that are facilitating long-distance trade, trade patterns and introduction into new areas is heavily reliant on membership of these kinship groups. State societies generally deemphasize these kinship groups, eventually doing away with them like in modern society, but the mechanisms for developing social trust are based on a very centralized development of the legal system--the government essentially enforces that people have to be trustworthy, or they face the wrath of government otherwise.
Essentially, in short, the idea that humans have regularly been trusting people based on asking their close compatriots who they can vouch for doesn't seem to accurately model how trust develops in societies for most of human history--you either have vouching based on direct observation (in small societies), vouching based on identity (larger non-state societies), or centralized authority vouching for people (state societies, including today).
Also, the interpersonal web of trust is IMHO quite clearly failing in the age of mass media, and not just recently.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I don't know how we fix this, but it does seem like a big problem.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
You mean until the DNSsec key rollover completes and they're no longer in that role?
As opposed to what we have now, an indefinite period where crappy non-updated devices will still accept these certs and nothing can be done about it?
The Web PKI works great for browsers. Everything else, not so much.
(To be fair, the crappy devices¹ would probably fail to deal with the periodic DNSsec root key rollovers. But at least that's a fail-closed.)
¹ a label that includes older versions of systemd-resolved, for whatever that may mean
That's not to say that we can dispense with these institutions, but they are not whence the trust comes, rather they're a sort of nucleation site around which the peer to peer trust we've been relying on for hundreds of thousands of years can coalesce. The institutional structures are an index of things that we may or may not trust, their utility is that the index lowers the information processing burden of doing so in either case.
The friend-or-foe decision started to become computationally burdensome as populations increased, and it has indeed failed to scale, but it's still in there doing its best, making it look like the centralized attempts are working because often they're aligned.
But when the people in charge decide that they no longer care to protect the people who they're in charge of, that alignment slips, and suddenly the most powerful organization on the planet can't win a war against pond scum.
The trust has always been peer to peer. We can find ways to improve it, so it works better for us, or we can continue the charade that it's replaceable with a central authority, to the peril of those who embrace the substitution.
Ultimately this is a synchronization problem. EMudhra has lost their standing in the web of trust we've been using all along, and because the path between them and the average user is so circuitous, third parties need to step in and revoke that trust in the centralized trust stores that we're using to approximate that web. If our technology better aligned with how our actual trust worked, those paths would be short, blast radii of this kind would be small, and the urgency of action could be assessed more accurately because the actors would be protecting themselves from risks they understand, rather than acting to protect users that they don't know and who don't know them.
Visibility through Certificate Transparency logs would at least make it a matter of record that it occurred, though.
I agree, you can't kill a TLD operator. But WebPKI relies on TLD operators to establish identity, so I don't see what we gain from WebPKI, other than mandatory CT logs, and not having to faff about with DNSSEC :P ... both of those are pretty valuable, of course.
I'll put it another way: why do we want a CA to tell us that someone controls a domain, when the TLD could tell us? The TLD is authoritative with regard to who controls a domain; CAs can only report on what they have observed.
Maybe if people implemented name constraints and the CA/B forced CAs to use them. But not as it is; unconstrained sub-CAs are a scourge on the Web PKI.
eMudhra is not such a case. They are a root CA that issues certificates to the general public all over the world. In fact, they are now the preferred vendor for numerous US universities via InCommon[1]. There is no way to meaningfully name constrain them.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
It can also be disabled at the whim of anyone operating a resolver, as was seen with Cloudflare during the .de TLD DNSSEC snafu back in May.
This particular issue has nothing to do with "aligning authentication with the name hierarchy". The exact same thing that happened to EMudhra could happen to any TLD operator. The difference is that we have essentially no recourse when that happens, and far less ability to detect malfeasance.
What is a difference, versus DANE, is that TLS uses CT. Compromised TLDs happened recently but was found out through CT. A compromised TLD still kills the whole system for a time regardless.
I think DNSSEC still has some use in enforcing email TLS and for CAs.
Interestingly, InCommon used Sectigo prior to this year. I'm guessing eMudhra costs less.
In the main, attacks on the DV process aren't protocol attacks; they're simple ATO attacks on registrars. The DV part of the WebPKI is certainly gross and could be improved, but evidence from the last 20 years says pretty clearly that the DV process itself is not the most urgent risk in the system.
My first hope is that we continue to ratchet security requirements for CAs upwards, and in doing so wash more and more incompetent operators out.
I don't care if you write 100,000 or 1,000,000 words: there's nothing wrong with DNSSEC or security that rides on it in principle except that it distributes trust decisions that have been centralized in those objecting to DNSSEC.
It is very bad for the internet to depend for its security on a tiny number of individuals to make global judgements about who's a bad actor.
You said “no recourse,” which I took to mean no recourse even if known. I don’t follow that. The key/DS record at root can be changed. There may not be a “TLD death penalty,” but the key could be changed.
Now without CT it may be tougher to know about the compromise. For websites that don’t have the luxury of certificate pinning, a compromise of TLD kills the whole thing, at least for a time.
The "recourse" issue is that after you resolve a DNSSEC misissuance, you cannot then murder the TLD operator that did that to you. But that's exactly what's about to happen with EMudhra.
As far as I know, no TLD operator has gotten the death penalty. It is technically possible for root to change the IP addresses, but it would be bound up in whatever the governance is of the domains.
But the downside isn’t something new that would just apply to a DANE world. That downside still exists in the CA world. The CA is just an additional point of failure, along with the TLD.
What is added by the CA world is CT.
In a DANE world, I need only trust the DNS authorities, and there are ways to boost trust in, e.g. the dot-com key management. (Could even go full ring signatures if we wanted.)
Someone over on foo.wtf.lol can't hurt me.
Meanwhile: this CA is likely to be killed outright for this fuckup.
This way, fuckups like this CA can't hurt me because they couldn't break DANE and IC couldn't hurt me (more than they can already) because WebPKI would have to approve too.
I don't think the CA death penalty addresses the problem. Just as in the physical world, the death penalty isn't a perfect deterrent.
If you care this much about your root set, consider having Claude roll you a Chromium or Firefox with a user-controllable set of roots that you ruthlessly prune and manage. Unlike DANE, which simply isn't going to happen for a whole variety of reasons, your personal root program (I gift that term to you if you want the coinage) actually could help people with your preferences.
The private key for all 8 "compromised" CAs is a single byte: https://mastodon.catgirl.cloud/@49016/117474217042191790
One of the private keys is literally "4".
I'm putting "compromised" in quotes here, because I can't call it that without cringing and cursing.
[cf. also: https://news.ycombinator.com/item?id=50169109]
edit: The reported private scalars were 4, 56, 66, 115, 155, 156, 230, 237 across the eight certificates so its unlikely i will win my bet :'D
The plot thickens.
Broken HSM? Misconfiguration? Confusion? Straight up lie?
> Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
…
Ah yes.
"The system says something is wrong, but it's also 4.30PM. Let's just re-sign the thing and go home."
Good luck recovering from this.
That seems like a WAY bigger problem than TLS, but AFAIK Windows still validates code signing certs against CRLs before trusting them (can't remember if it soft-fails).
One of the issues with sub-CAs (well, X509 certs in general) is that they can be ghosts in that there's no way you'll ever get it as a response on a TLS connection, so automated scanners on that won't find it.
CT should cover this (i.e. expose the cert so you can check for weak keys), but I have no idea if/how CT works for sub-CAs. [ed.: see below, doesn't seem to be in CT]
[ed.: these are technically sub-CAs, but rather cross-signed for bootstrapping a new root CA. Thanks @agwa for pointing that out.]
They had one job, and they blew it. The system is based on trust and they've lost it.
I'm going to remove their roots from all my stuff in any case.
key_string = to_string(*key);
key_string = to_string(key);Given that 4 is the value shown in this well known XKCD panel, it almost seems like a (im)practical joke or trolling.
Oh no, they reverse engineered the random number generator!
Compromised _keys_ not compromised CAs:
"The private keys of eight eMudhra intermediate CA certificates have been compromised."
"The private keys for the intermediate certificates were incorrectly generated by the CA."
The certificates with incorrectly generated keys have been "revoked" but will the eMudhra CA remain in business
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
[1] https://blog.mozilla.org/security/2015/03/03/revoking-interm...
[2] https://hacks.mozilla.org/2025/08/crlite-fast-private-and-co...
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
They don't blame the HSM.
Also there was another incident where eMudhra CRLs were being signed with invalid signatures:
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
This bit sets my hair on fire. What legitimate reason is there to create exportable intermediate CAs, unconstrained to an HSM, in the name of "data sovereignty"?
> I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
__
Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
Someone actually qualified please give credible answers. I'm just pretending.
I was not expecting to find a silver lining _this bright_ but you have a point.
Apparently it's even worse
Also who hasn't left their private keys hanging around web-accessible, eh? lol
Private keys can be trivially recovered from the public keys because they're tiny. I found D=156, flüpke found D=4.
Yes, that's the entire private key.
4, 56, 66, 115, 155, 156, 230, 237
curl https://example.org --connect-to example.org::2.29.126.86:
curl https://q3k.google.com --resolve 'q3k.google.com:443:135.181.235.221'
Thanks to mastodon user q3k@hackerspace.pl.
Rust with rustls? Upgrade to rustls-webpki 0.103.16 and/or webpki-roots/webpki-root-certs 1.0.10.
Python? Upgrade certifi to 2026.10.19.
Homebrew ca-certificates? 2026-09-25_1.
Electron? 44.9.1, 43.8.1, or 42.11.16.
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
With that being said, that Bugzilla is the best thread to understand "how the fuck did this happen?"
In that case it was ssh sessions being intercepted.
But my question is if a tor exit node or a vpn provider or your internet provider has a private key to any CA then can they mitm any normal web traffic even though it will show up as https without warnings in your browser?
When you're using a modern browser that browser will require any certificate to have a certificate transparency log stamp. Certificate Transparency logs are public append only logs of issued certificates.
So you access example.org. Your ISP uses the compromised CA certificate to create and sign a rogue certificate for example.org. It now has to submit this certificate to a public certificate transparency log and gets back a proof for that. Now it can use that rogue certificate and the proof that it's in a CT log to impersonate example.org and MITM your traffic.
CT does not prevent compromise, but it makes it far easier to detect. If the people behind example.org have a look at the CT logs they will see the rogue certificate there that they did not request. This makes rogue certificates and by extension CA compromise detectable. So your ISP cannot just do this forever in secret.
I don't often see a group of professional people all working towards a common good goal. (Which here is simply to revoke the certs from everywhere and make sure everyone has the information, but it still makes me happy)
> This would have been possible since the certificates were created in April 2025. ... CT-enforcing clients like Firefox, Chrome, Edge and Safari could not have been attacked.
Meanwhile, back in April 2025: "Is it possible to enable CT checks in curl?" "No." https://github.com/curl/curl/discussions/17078
> No. Nobody has added any such functionality (yet). Feel free to work on that.
If someone thinks a feature can be useful, especially in the era of agentic coding, instead of asking, do it.
This thread (linked by the page) seems like a better source: https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Ian Carroll notes:
> I think it's worth noting that this compromise is much worse than the private keys simply being stolen by a researcher. They are inherently weak single-byte keys that anyone can factor, and the issue cannot be contained.
Yeah that’s just oof. Not hard to imagine the kind of code error that could cause this, but hard to imagine that any serious vendor of cryptography hardware or software could make a mistake like this.
I've only found one other person talking about this online.
What's very strange is that we don't use Godaddy directly and it's signed by Starfield G2 (now R1) - not the Godaddy root CA.
Has anyone else had the same communications around their TLS certificates?
Your certificates should never be signed by the root - this is prohibited by the CA/B Forum (see Baseline Reqs 6.1.7). My understanding of best practices is that the root signs intermediates then is taken offline ASAP, and the intermediates sign leaf certificates.
Re: Starfield, GoDaddy (and AWS) use the name in trade but operate the issuance. The dev-sec-policy mailing lists should detail any transfers of ownership (in which cases, it is common to maintain an old brand name under a new root).
- Could this be used to run a benign mitm attack on the traffic of some apps, consoles etc? I could see some interesting uses for reverse engineering. For example you can't simply add a CA to the system store on Android anymore. But this one is already there. It won't work for long but it could be used on any unpatched client.
- Can't we just revoke the entire CA? Just like what was done with Diginotar. Edit: apparently they just revoked the intermediates.
- Why isn't there a mechanism where a domain owner can specify which CA it allows? Similar to SPF which allows you to specify which mail server can represent you. It's kinda insane that foreign CAs can just fake anyone's certificate.
CAA records do this, but only for issuance, not for use, so it only protects against eg. unexpected domain validation, because the CA has to respect the CAA record. Extending that to use seems like a logical step to me.
One issue might be that from what I understand Android itself also enforces CT, but it has fail-open design if it cannot obtain the updated list of qualified CT operators for 70 days. So you would need to block the access to that endpoint.
This is assuming the CT logs will stop/stopped actually giving SCTs for those CAs.
There is DANE (TLSA) but seemingly only supported by e-mail software and not web browsers.
They are certificates of authority. They have authority, or are given it, and that means don't accept them.
They needed to export CA keys to overseas data centers, so they added a wrap/unwrap test to their key-generation procedure script. Somehow that generated a second set of weak (test?) keys from the CryptoServer Se-Series Gen2 HSM.
Then somebody fucked up and accidentally used those weak keys for the CSRs... then their root and Sectigo signed it. And it shipped.
That's why the private keys are literally one byte.
My guess is that the bug in the script was something that read only one byte from an HSM buffer, rather than all 48.
Particularly concerning is India's UIDAI https://lea.uidai.gov.in/
The intermediates seem to be difficult in that the root stores don't seem to have a mechanism for it (… kinda seems like an oversight, at this 20:20 vision point?). If eight of the CA's intermediates are hosed … how many are still "good", and are they good good, or are we just fooling ourselves?
LE has two EC intermediates, by comparison.
sudo certctl untrust /usr/share/certs/trusted/emSign_ECC_Root_CA_-_G3.pem
=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3
=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign Root CA - G1edit: Disabling all the eMudhra CAs in Android settings does nothing, cool?
edit: I guess there's no way to manually ask for CRL update, cool??
edit: Gave up and reinstalled Firefox. Seems to be fine now.
https://discourse.ubuntu.com/t/an-update-on-upki/77063
My interest in building OneCRL support into upki has definitely increased this week. Issue:
https://github.com/rustls/upki/issues/229
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
One blocks everything; one blocks 6 out of 12; one blocks the other 6 out of 12; one blocks nothing.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
It's cute you have a magic DIY CRL. You accepted these assholes into your root program.
They lost the KEYS.
What is the screening process for CA's that can't be bothered to use a HSM?
Even Honest Achmed wasn't this careless.
it's done in the open, and you're even allowed to voice your cute opinion during the multi-week public commentary periods.
here is an overview https://wiki.mozilla.org/CA/Root_Inclusion_Overview
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
However practical applications do care, and so e.g. the "Debian Weak Keys" are a relatively small set of keys which would have been generated if you used a particular patched OpenSSL to generate "random" key pairs after a mistake patched out most of the randomness. Many of them were used by Debian sysadmins particularly in the window between the patch and the discovery - and so all CAs are expected to check that you're not trying to get a certificate for the public half of one of those pairs.
There's nothing "bad" about those keys except that well, we know some people used them so bad guys will try the private halves of those keys just in case. Likewise there are a few apps which shipped with a baked in "default" private key, and all the paperwork to apply for a certificate but obviously a user may not realise that if they just fill it out with their name and details, they're now vulnerable to attack by literally any other user of that software. They were supposed to generate a random key pair and re-generate the paperwork for the resulting random values. If they don't realise we need the CA to say "Nope, that's a bad idea, not issuing" even though these keys aren't special in any other way than they're given to every single user of that software.
IIRC when I worked for a famous US corporation their middleboxes relied on such a well-known key and at some point I asked "Hey, this seems wrong. Why did you do this?" and as often has happened in my career I got no actual reply to my question but magically the issue was fixed in short order. The first clumsy attempt to fix it broke some things, but they gradually figured it out. Glad I could help, I guess.
Edit: however on Firefox Focus 157.0.1 they are blocked.
Just another step towards a tiny number of browser vendors and infrastructure companies controlling the internet.
I fully expected disabling them in the phone settings and rebooting would have fixed it, but no?
Don't really wanna reinstall FF and lose all my configruation.
Edit: I see the below paragraph is quoted extensively below. I hadn’t seen those comments yet. Oh well
Source: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c11
Content:
We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted. Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).
and thx for the context. i rly wonder now how it went down lol
> The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order.
That is within the realm of believability.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Nobody thinks this is a good thing. It's bad. Hopefully, eMudhra gets distrusted. We've done that to bigger CAs.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
This is absolutely false, it's a simple policy change to enforce code signing, any organization that is serious about security can do it.
I'll give you the benefit of the doubt and assume you're another Linux guy who doesn't understand Windows security.
The concern of course is with malware being signed, which is far more likely to be abused than some TLS MITM fantasy.
You already need compromised systems to install malware, signed or not. Not a real problem.
For TLS MitM you just need to use a shared network, wired or wireless.
https://groups.google.com/a/mozilla.org/group/dev-security-p...
2026-10-17 20:18 - Full CCADB check finds the cross-signs by Sectigo are also affected
Inclusion in the Chrome Root Store requires that a CA Owner continuously demonstrate the technical competence and operational rigor necessary to protect Chrome users. Due to the seriousness of this incident, the Chrome Root Program is conducting a thorough evaluation regarding the status of eMudhra’s remaining CA certificate to ensure these standards continue to be met. We will share further updates as our review progresses.
from a post on https://groups.google.com/a/mozilla.org/g/dev-security-polic...
I can't imagine one part of a business being on the up and up, and the guys down the hall running completely rogue.
The whole point of a key ceremony is that you're not going off-script and YOLOing random crap.
https://utimaco.com/news/blog-posts/global-encryption-day-en...
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
And THAT is where I'm drawing the line between "honest mistake" and "gross negligence". For that to not raise giant red flags is an unacceptable failure in process and work culture.
I'm not saying this will never happen, just that rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation. I fully expect IC agencies to get some CAs killed and be fine with that.
I bet you have at least one copy of OpenSSL on your Windows system, probably multiple. Using the Windows native APIs is work. OpenSSL just works everywhere.
(Also, "the platform" doesn't technically even provide the cert bundle, depending on exact distro & packages.)
How you have that root cert on your phone with a 3 year old CA set I have no idea, presumably something updated your CAs in the meantime? (Also, aren't you incredibly exposed to the general onslaught of vulnerabilities from the past 3 years?)
> The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant.
A FIPS HSM does not forbid one to import keys into it.
You trust the magic black box far more than I do.
My big fear about HSM is that they are inscrutable by design. There is no way to verify they are working as advertised. Plus side, all sorts of fancy hardware to protect your keys. Down side, all sorts of fancy hardware that obscures flaws in the implementation.
https://ctlogs.io/search?mode=fingerprint&q=42c4accd12c4f595...
https://ctlogs.dev/search?q=42C4ACCD12C4F59597925E1A08A7A901...
https://ctlogs.io/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B3...
https://ctlogs.dev/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B...
Nothing for "CN=EM DV TLS CA - G3B,O=eMudhra BV" (not sure which sha256 this uses, 2nd is the spki sha256)
I should be clear here my understanding is/was these are sub-CAs. Apparently they're (trying to be?) roots themselves? The point of logging a root cert in CT is somewhat moot… though tbh I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
And it is, or at least, was. The CT logs when I last spent too long staring at them were complete indefinite logs. These days most (all? almost all?) are time limited so e.g. there'll be Clown 2026 and then Clown 2027 and a certificate is logged to the CT log corresponding to a not-after date IIRC. This way you can throw away a log of certificates expiring in 2025 a few weeks into 2026 because who cares, they've expired, and so they don't just grow forever.
I no longer have a role where I care a great deal about this for my day job, and I do not know how to ask ctlogs.io for the thing I want. So probably this is still logged but I couldn't promise you without getting back into this. Maybe at the weekend.
> One of the issues with sub-CAs (well, X509 certs in general) is that they
> can be ghosts in that there's no way you'll ever get it as a response
> on a TLS connection, so automated scanners on that won't find it.
My understanding is that if something, like a browser, connects to a SSL site, they will either have or can obtain, one way or another, a complete chain all the way up to the root, so there can be no "ghost" certificate. And I would imagine a scanner could do all those same steps, and thus access all the same certs.
Is this a deficiency in scanner software? You seem to indicate it is a fundamental weakness in X509; I'm just trying to understand what.
Could you maybe describe a specific example of what you're describing?
More realistically, especially with sub-CAs, they might only be used on e.g. corporate networks. TLS scanners may not be able to reach any of the hosts that uses such a cert. As the sibling post points out, clients checking for the cert in Certificate Transparency logs counters that (adding a requirement that the cert be publicly known, which TLS/X509 by itself very much does not have) — but of course this only helps if your client does CT…
VPNs, embedded software (IoT, automotive, aerospace, medical, industrial etc.), business systems, you name it.
Given Wayne's hints in the timeline that this issue was discovered by an agent, it might be one of the first big corporate death sentences due to AI?
The problem is the death penalty case is aimed at the wrong target: The entire Web PKI system remains dumb and bad, and the correct answer is to dissolve the CA/B and find people with practical security experience.
[0] https://emudhra.we-have-a-problem.com/roots.json
[1] https://www.entrust.com/company/newsroom/entrust-sells-publi...
Issuing sub CAs to non-root operators has historically been a big loophole though it's not as bad these days since multiple root programs now require approval before CAs can do it.
Now it's coming out these guys used "4" as a private key. That's a crazy amount of incompetence and malfeasance. Did they vibe-code a CA? HOW DOES THIS HAPPEN? I am dumbfounded.
You couldn't even do this if you installed Microsoft Small Business Server 2008 today, unpatched, with the CA on default settings.
Obviously the system failed here, but it's a failure of a root certificate operator, not a sub-CA operator as claimed by eqvinox.
(In fairness this is a self-fulfilling prophecy: a lot of validators handle NCs poorly or not at all, so there's no impetus to rely on them. But that's not something we can really save in the Web PKI's current state.)
I agree with your assessment but it's not an excuse to give up without trying. I hope this is just because I'm not looking in the right places, but I don't see people trying.
What really makes me salty is that there was enough energy to mess with the rules around EKU, the primary effect of which is that non-browser software is now starting to just ignore the EKU bits. Even if it would've amounted to nothing, I'd really rather that effort had gone into NCs.
They hold intermediate certificates issued by TWO different root CAs, neither of which they operate.
So two separate entities didn't bother to read their certificate policy or inspect their operations. Have we not learned anything from past mistakes?
Roots should be responsible for all their sub-CAs, full stop. If they don't trust them enough to accept that responsibility, then the system is broken.
Let's just say they've had issues with signing sub-CAs before where those sub-CAs went on to misuse them before… looks like it was around 2011?
This is probably a rough definition for "Entrust was banned from being a CA and sold the business off to a competitor". In part, because I also do not think the root programs would just allow Entrust to continue operating the same business under another brand name, unless they are even dumber than I already thought.
The Root CA must sign requests after authentication, but a "manual" approval is also accepted if done with the appropriate oversight with witnesses (clearly not the case here).
(I really am salty about the EKUs. In a few years the EKU field will just be useless, due to operational annoyance driving devs to ignore it. Except in browsers of course.)
(But in any case, CT ensures that it can’t be “quiet.”)
1. CT does not do anything if a state actor demands a CA in its borders generate a cert without publishing to the CT logs (or a three-letter-agency doing this illicitly). They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees.
2. Browsers won't check for SCTs in certs if they come from a private CA. you can MITM any domain through ZScaler all day long and nobody will notice
3. Wildcard certs make it easy to make/renew a cert that looks normal but hides requests on new/different hosts
4. Non-Browsers do not validate SCTs
I don’t think you can get the SCT from a log without the log publishing it? That would defeat the purpose.
“Certificates that support certificate transparency must include one or more signed certificate timestamps (SCTs), which is a promise from a log operator to include the certificate in their log within a maximum merge delay (MMD).[4][3] At some point within the maximum merge delay, the log operator adds the certificate to their log.”
> Because no TLS certificates chaining to these six intermediate CAs were logged to CT prior to our CRLSet block on October 17, no fraudulent certificates issued by these CAs could have been trusted by default in Chrome or other CT-enforcing browsers.
Well you can, but only if the log cooperates.
That is, the SCT is merely a promise to add the certificate to the log by some particular deadline. Nothing technically stops the log from disregarding that promise. But if it does fail to log the certificate, the SCT itself is proof of its misbehavior.
And it's not enough to subvert a single log, because browsers require SCTs from multiple distinct operators in order to consider a certificate valid.
The lack of CT entries only proves that (most likely) browsers were not MITM’d.
Basically no one outside the major browsers verifies CT logs, so you can still attack a lot of people even without publishing.
CAs publish to CT logs. If an attacker has the private key, nothing stops them from signing certs without the involvement of the CA or CT logs.
It's all rather hypothetical and unlikely, but you could certainly try attacking something like downloads.openwrt.org¹ like that — especially if you can tell apart the TLS client in opkg from a normal browser.
(¹ luckily, OpenWRT and most other package repos don't actually rely on HTTPS for package integrity and authenticity. And OpenWRT would be the most susceptible due to the "exotic" TLS client stack, if any.)
But if you only want to MITM the connection used by 'pip' to install python wheels - that doesn't check for CT records at all.
No, the cert can be issued with no (or invalid/junk) SCT and will work fine with clients who don’t check SCT, which is most non-browser interaction.
Am I missing what you mean by monitor?
My point is more that as a user of a high-level library like reqwest, I'd like to talk safer defaults when root CAs are in use