Ask HN: Cloudflare broke my domain's DNSSEC making it unreachable since 4 days

139 points by medguru 4 years ago | 119 comments

tl;dr - Cloudflare rendered my domain inaccessible and support has been ignoring the ticket for 4 days, what's the fastest way to get technical assistance when on a free plan?

Last week I transferred a domain used for a personal project from my old registrar to Cloudflare. After the transfer was finalized and new NS records had propagated, everything resolved normally and everything was working fine. I then enabled DNSSEC, and after a while the domain would no longer resolve. Every DNS server I try - Google, Quad9, OpenDNS, even Cloudflare's own DNS on 1.1.1.1 - returns SERVFAIL. The excellent diagnostic tool on dnsviz.net tells me that the domain is returning bogus DNSKEY/DS/NSEC responses and bogus delegation status. "no SEP matching the DS found".

I tried canceling the DNSSEC setup and waiting for over a day, with no effect. I re-enabled DNSSEC setup and waited for 3 days, with no effect. Cloudflare's control panel has since several days now been saying that DNSSEC will be enabled "in the next 24 hours". My site cannot be reached, and Cloudflare's support cannot be reached.

I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name.

How can I get their attention without paying for an Enterprise plan? I would like to think that basic functional service should be accessible even when using Cloudflare only as a registrar with fundamental DNS on a free plan.

elithrar 4 years ago |

(It sucks that I had to see this on HN)

Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?

medguru 4 years ago | |

Thank you for the attention, e-mail on its way.

elithrar 4 years ago | | |

Just replied.

Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those).

For others: I'll let the OP share what details they would like to, as this is their domain.

gigatexal 4 years ago | | |

Come back and tell us what happened and the resolution.

quinncom 4 years ago | |

Is there a way to pay for priority support while on a free plan?

I'm nervous about migrating a domain to CF which is used for glue records, and want to have immediate support access if something goes wrong.

elithrar 4 years ago | | |

The $20/mo Pro plan includes formal support (one of the big perks of Pro!)

medguru 4 years ago |

Update for those who were curious:

Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again.

Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive.

If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort.

If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well.

I've been OP and this has been an update about my domain woes.

Yeri 4 years ago |

(CF TAM here)

All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though).

The correct procedure would be:

* turn off DNSsec on old registrar (and wait a day or two)

* update NS and/or migrate domain

* wait a while and make sure it works

* turn on DNSsec in CF dash and update DNSsec settings in the domain

It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.

mike_d 4 years ago | |

The correct steps are to completely disable DNSSEC and remove the relevant records. It is far too fragile for the majority of use cases. Effort should instead be put into deploying things like DNSCrypt that implement transport security and confidentiality.

Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much.

Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.

cryptonector 4 years ago | | |

I want both, DNSSEC and DNSCrypt.

WebPKI is a joke because it lacks name constraints and so isn't and can't be hierarchical. DNSSEC is a true PKI -- you can still have multiple roots if you like and don't trust ., but it's got name constraints, so whatever domain you graft an alternate PKI at, from there on down you get bound to that PKI. This is really, truly fantastic.

Add DANE and you have a complete replacement for WebPKI.

DNSCrypt is needed to increase confidentiality, it's cheaper than DNSQuic and such things. Unfortunately .'s and com's and major TLDs' NSes are unlikely to want to waste CPU cycles on any DNS confidentiality solution, and even if some TLDs did, unless clients use QName minimization, users gain no confidentiality -- . and the TLDs all have to adopt it.

The two, together, would be truly fantastic.

mike_hearn 4 years ago | | |

DNSSEC is however the only way you can make TLS really work. The whole TLS ecosystem is dependent on CAs that are basically just a giant hack. They're signing a statement that they did a bunch of DNS resolutions at a point in time from different network vantage points (maybe, hopefully), and got consistent answers. DNSSEC+DANE lets you get the actual data you want (domain name->public key binding) from the root source, without needing the complicated middlemen.

Spunkie 4 years ago | |

I mean when you are adding a new domain, CF can clearly tell if a domain already has dnssec on or not. Seems like something that should raise a warning to the user.

medguru 4 years ago | | |

That would have been very helpful. The [Enable DNSSEC] button in the control panel is very assertive and confident through its casual and innocuous appearance.

Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.

_wldu 4 years ago |

DNSSEC is notorious for breaking things [1]. I use it on most of my domains, but I would not just 'enable' it on a domain that I cared about and that had real users without a lot of thought and planning. Nor should you.

[1] - https://ianix.com/pub/dnssec-outages.html

medguru 4 years ago | |

I figured it would work since there were no problems for the handful of my domains using DNSSEC with the previous registrar. Maybe the button should come with a warning label. I'll certainly be a bit cautious from now on.

paulnpace 4 years ago | |

> DNSSEC is notorious for breaking things

My understanding is that people break things.

ceejayoz 4 years ago | | |

https://en.wikipedia.org/wiki/Just_culture

> Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?".

Prominent (and very effective) example: Aviation safety.

DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/

snowwrestler 4 years ago | | |

Sure, everyone makes mistakes, but there’s still a difference between juggling rubber balls and juggling knives.

scrollaway 4 years ago |

> what's the fastest way to get technical assistance when on a free plan?

Upgrading to a non-free plan?

You don't have to upgrade to enterprise, but even their $20/mo plan comes with support.

(Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)

tmikaeld 4 years ago | |

Why was enabling DNSSEC a bad idea? Clearly the origin registrar isn't handling DNSSEC requests properly, but the OP should still be able to revert to non-DNSSEC without issues.

scrollaway 4 years ago | | |

This post makes a better argument than I could write. And nothing much has changed in the seven years since. https://sockpuppet.org/blog/2015/01/15/against-dnssec/

nemothekid 4 years ago | | |

>but the OP should still be able to revert to non-DNSSEC without issues.

Slack had a 24 hour outage doing exactly that; so I don't think "revert to non-DNSSEC without issues" is trivial at all.

pixl97 4 years ago | | |

Wasn't there a thread on HN within the last couple of months that says switching back when things go wrong is actually very difficult? Intermediate servers or something cache the DNSSEC issues and things tend to break for 24 hours at a time. Unfortunately I can't remember the details.

medguru 4 years ago | |

Can you please explain why DNSSEC was a bad idea in the first place? It worked perfectly fine with the old registrar.

jgrahamc 4 years ago |

Reading this hurts. I see that @elithrar has given out his email address and is following up but I will also be following this internally to understand what happened.

hericium 4 years ago | |

Since the issue was made public, we'd love to see some details in case OP (throwaway) doesn't come back.

medguru 4 years ago | | |

I'll be back.

groffee 4 years ago |

> How can I get their attention without paying for an Enterprise plan?

Just comment on HN and they'll crawl out of the woodwork.

paulnpace 4 years ago | |

Also, the community forums are generally quite useful when something isn't working. I've posted there with an actual problem maybe twice, but the problems were resolved within 30 minutes.

medguru 4 years ago | | |

Which is a bit strange. I would have guessed that the support ticket system would be prioritized by staff.

wnoise 4 years ago |

English language usage note: "since" takes a past point in time, not a duration. You want either "unreachable for 4 days", or "unreachable since 4 days ago".

andrewstuart 4 years ago |

I had the same problem.

I registered a domain at Google Domains.

Then I configured the domain at CloudFlare.

At first it worked OK then I started getting SERVFAIL.

I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK.

Cloudflare was not at fault in my case.

oneplane 4 years ago |

> How can I get their attention without paying for an Enterprise plan?

By paying for the cheapest plan, or any plan at all for that matter.

redm 4 years ago |

Enterprise plans no longer come with "premium" support either, you are looking at 20% over contract value to get a similar level of previously included support and an SLA. To be fair, CloudFlare provides a lot of services for free and $20 premium plan with upgraded support seems like a pretty good deal!

medguru 4 years ago | |

For any form of business endeavour that would be the obvious lowest starting point. For private users a $20/mo. price tag for registering one or a few domains would turn effective TLD pricing on its head. Suddenly owning a single .net domain is no longer $20 per year like it is with all the other thousand registrars, but instead it would be $260 per year.

Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.

pteraspidomorph 4 years ago |

I had a problem with a similar effect some time ago but I run my own DNS (no Cloudflare). I accidentally clicked a button in my control panel to regenerate the zone keys, which means the published keys mismatched the new zone signature for a couple of days until I was able to get the registrar to update them and everything propagated (even when a registrar supports the .eu TLD they are usually severely lacking in automation). The control panel devs have since added a confirmation dialog!

InitialBP 4 years ago |

Sorry to hear about your problem, a quick recommendation would be to keep the temporary DNS name you bought and simply redirect to your previous name once you have the issue resolved (or vice versa if the branding is less important to you.) This way your users won't need to know or care about the change anymore aside from this temporary setback.

williamtwild 4 years ago |

>I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name.

If this was that important then you should not have used the free plan.

medguru 4 years ago | |

Why do you presume the issue would have gotten immediate attention for the sum of $20? Customers don't make Cloudflare's terms, and customers didn't decide for Cloudflare to offer a free plan with zero markup for their registrar operations.

There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.

Spunkie 4 years ago | | |

They are also the only domain registrar that I've interacted with this decade that does not allow you to set your own nameservers. Effectively locking you into using cloudflare DNS and related services until you can transfer again.

Its frankly, disgusting.

ejjpi 4 years ago |

I'm also noticing that Cloudflare support is going terribly downhill.

I have an issue with the Cloudflare infrastructure on my domain since WEEKS, giving me thousands of 503 Service Temporarily Unavailable errors per day (cloudflare side, not the origin server) and nobody seems to care or able to resolve.

Removing the ability to create support tickets on free plan doesn't help at all, I mean, I get it why they're doing it, but asking on their community forum as an alternative it's not an acceptable solution. Neither going after Cloudflare employees on social media platforms hoping for a reply.

If I'm also going to pay for their services such as Zero Trust, domains registrar and R2, why do I have to switch to a Pro plan just to open a support ticket? Perhaps a middle-ground solution like 1 free support ticket per month on a free plan would be a good compromise?

I still think they're giving an incredible service and value for free, but this sucks.

warrenm 4 years ago |

First problem - trusting Cloudflare :|

I've had nothing but problems with them personally

I know some people swear by them ... I'm in the "swear at them" camp

xbar 4 years ago |

dnsviz.net is awesome.

jSherz 4 years ago |

Does your TLD definitely support DNSSEC?

medguru 4 years ago | |

Yes. It had DNSSEC enabled for over a year when it was with the old registrar.

phillipseamore 4 years ago | | |

You would usually need to disable DNSSEC, wait 24h, transfer, and then wait for at least 24h before enabling DNSSEC again.

b3lvedere 4 years ago |

-- removed. My apologies. --

medguru 4 years ago | |

My domain and users have ended up in limbo beyond anyone's but Cloudflare's control. I cannot transfer it back to the working registrar, or I would without being "angry at some free service". Why do you think berating me with snide remarks is helpful?

b3lvedere 4 years ago | | |

My apologies if you read it as a snide remark. I'm usually baffled when support is demanded on free stuff, but i see your point in this particular scenario.

codatory 4 years ago | | |

When it comes to dnssec, the keys are handed down through the same layer of delegation that gives you your nameservers. This is out of Cloudflare's hands as well, you have bad data up through the registry and out to (your tld's) root nameservers. Just like changes to your NS record delegation don't take immediate effect, these keys don't take immediate effect either.

It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.

Copenjin 4 years ago | |

If free implies "if it breaks don't ask us anything, hope it works" they should state so when you signup. Pretty sure it's not the case.

megous 4 years ago | |

Not really a service if it doesn't work. :)