Akamai Edge DNS was down(edgedns.status.akamai.com) |
Akamai Edge DNS was down(edgedns.status.akamai.com) |
This time i think /r/sysadmin pegged the issue first, great sub.
It’s just a completely random DNS outage, nothing more.
archive.org seems to indicate there was never anything there...
EDIT: So HN can't even take a joke after this? [0]
Either way, the joke's is now on the HNers in that thread.
How am I going to sell my AMC stock...
https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...
That's weird to me. I have been working in sysadmin/DevOps for over a decade, but it did not take me very long to learn that DNS outages cause massive problems.
Love this.
So many sites down... and unfortunately not one of them is Twitter
Reference: #11.453a2f17.1393u44848484.3aee33433
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
Both Bitwarden and 1Password are great.
I wonder how much they spend on multi-AZ redundant architectures...
Whatever multi-home means, why can't there just be one service provider that does that? And are we sure that these service providers aren't already doing that as best we might hope for? (For instance, Amazon already has multiple zones, etc.)
I suppose the one thing this can't protect against is some sort of political (broadly defined) threat related to the company itself.
Many of these outages are due to pushing broken artifacts or configuration to production.
A single provider can pretty easily offer geographic or network topological redundancy, but administrative and/or technological independence is pretty hard to achieve in a single company.
Using multiple providers for fancy DNS, like only providing IPs that pass healthchecks or geotargetting users to datacenters gets pretty hard, because the different providers have similar capabilities, but no uniform interface, so you've either got to do it manually, or you have to build out your own abstraction that is probably limiting.
If possible, insourcing DNS makes the most sense to me, because if you can't keep your service online, it's not the worst if your DNS is offline; and if you can keep your service online, you probably won't mess up your DNS too badly.
Specifically, 1.1.1.1 provided bad addresses (as opposed to no addresses), and removing 1.1.1.1 fixed my problem. By then it had returned a bunch of bad addresses and I had to flush my DNS cache.
Server: 1.1.1.1 Address: 1.1.1.1#53
Non-authoritative answer: Name: newegg.com Address: 23.35.185.6
vs
Server: 8.8.8.8 Address: 8.8.8.8#53
Non-authoritative answer: Name: newegg.com Address: 104.80.92.252
104.80.92.252 is newegg.com
23.35.185.6 is a server that provides an error message.
So 1.1.1.1 lied. The proper response would be to reply "I don't recognize that domain". Instead it said, "yeah, I know that, its here..."
Newegg was not down, and when I got macos to forget what it had cached from 1.1.1.1 I was able to use newegg.com fine.
(a component of my consulting work is reporting to financial regulators for institutions)
As CTO of a bank, I wasn’t aware of this. So either we wasted a ton of money and time constantly upgrading redundancy and business continuity technologies to satisfy our regulators… or this statement could be mistaken.
So if everyone searched "is google down" and visited the link on downdetector that was returned in the search, that would add to the downdetector count for that site.
Downdetector doesn't actually know if the site is up or down.
Downdetector only reports an issue if a significant number of users are impacted. To that end, Downdetector calculates a baseline volume of typical problem reports for each service monitored, based on the average number of reports for that given time of day over the last year. Downdetector’s incident detection system compares the current number of problem reports to this baseline and only reports an issue if the current volume significantly exceeds the typical volume of reports.
https://www.speedtest.net/insights/blog/how-downdetector-wor...
e.g. dig @1.1.1.1 www.nvidia.com +trace
... various things from the root ...
www.nvidia.com. 7200 IN CNAME www.nvidia.com.edgekey.net. ;; Received 83 bytes from 208.94.148.13#53(ns5.dnsmadeeasy.com) in 35 ms
So the main DNS is fine, but it'll never get an A record because the last link in the chain is toast -- edgekey being Akamai in this case, but all CDNs do this so they can route traffic. Normally, this is a good thing so they can shift traffic within 30 seconds on their side. Unfortunately, it also means it would take nvidia an two hours to point away from Akamai.
So for example:
Top level domain for nvidia resolved fine..
dig @1.1.1.1 nvidia.com => status: NOERROR, Nameservers are ns6.dnsmadeeasy.com
But the website didnt. dig @1.1.1.1 www.nvidia.com => status: SERVFAIL,
The Nameserver for the this www.nvidia resolved to the akamai nameserver which had a problem..
dig @1.1.1.1 www.nvidia.com NS => CNAME e33907.a.akamaiedge.net.
Multiple websites including DraftKings, Airbnb, FedEx, Delta and others appear to be experiencing issues.
https://www.bloomberg.com/news/articles/2021-07-22/multiple-...
That makes me think that whatever the fix was, it had to wait for some one-hour cache to expire before it took effect. I'm very interested to find out what the cache issue was, more so than what the original bug was.
> So many websites are down, are AWS servers down or something?
> Amazon web services is down which is affecting a lot of company web sites and services. Not sure what is going on.
> Miss us? @aldotcom and a whole bunch of other folks have been knocked off the internet by what appears to be an AWS attack/system failure. We'll be back. ?
Basically soft-invalidate your local DNS cache but it back from the cache graveyard if DNS is down.
From what I observed here, it was more internal DNS related: Newegg was serving an opaque “DNS failure” error page from Akamai’s front-end which is likely because their infra was failing to resolve names internally.
Please keep comments like this off HN
GOV.UK for example uses both aws and gcp for DNS
But then, the cached values from AWS take a while to clear, TTL never seems to be applied properly. It always feels like the worst case in such a scenario is you can point everyone at the right thing within 24 hours.
What I'm a bit surprised / unsure of is what happens when I run "dig ns gov.uk". The results are:
gov.uk. 21559 IN NS ns1.surfnet.nl.
gov.uk. 21559 IN NS auth50.ns.de.uu.net.
gov.uk. 21559 IN NS ns3.ja.net.
gov.uk. 21559 IN NS ns2.ja.net.
gov.uk. 21559 IN NS ns0.ja.net.
gov.uk. 21559 IN NS auth00.ns.de.uu.net.
gov.uk. 21559 IN NS ns4.ja.net.
EDIT: I see that ja.net i.e. jisc.ac.uk "manages the second level domain .gov.uk" -- https://www.jisc.ac.uk/domain-registry . I imagine that uu.net and surfnet.nl are there for redundancy
Problems starts when you want to easy make frequent changes and introduce complex software to manage DNS zones (and complexity usually comes with bugs).
The whole reason it takes a domain 24h to fully work with DNS is because it propagates the information other DNS servers, thus making not be a centralized service.
Relatively short TTLs are ubiquitous these days though.
https://namebase.io is a "registrar" for it.
https://learn.namebase.io/starting-from-zero/how-to-get-a-na...
This is so convoluted it actually makes the whole thing a non-starter
What’s the single point of failure?
Services like Akamai use short TTLs for their edge services for a variety of reasons, not least because if one of their edge servers goes offline (for planned or unplanned reasons) it lets them sub in a new one and have it receive traffic immediately, rather than have a bunch of clients continue trying to talk to a dead node. So sure, you can increase those TTLs to trade 'what if the DNS server goes down?' risk with 'what if the edge server goes down?' risk...
But keeping the edge servers up and running is probably a lot harder - they need to scale more to handle traffic load, they have to actually handle client data, TLS termination, much more complex configuration.... so if I'm placing bets on which of those things is more likely to die on me, it's the edge node, not the DNS server.
Clearly a big one.
Should probably have a script play that on loudspeakers when monitoring detects problems /s
Probably reported Google as “down” because a whole bunch of people use the word “Google” when they mean “internet”.
dnsmasq has a really useful feature for dealing with this: --bogus-nxdomain
I did this with a website I liked which had let the domain expire. It worked for quite some time, until the VPS/whatever expired too. Good thing the Internet Archive is a thing.
Better alternatives are Cloudflare, Fastly, AWS CloudFront.
Google Cloud CDN always seems to have very good latency but a very bare bones feature set and no edge compute I can identify. Support is always a huge red mark for Google anything.
Content-Encoding should be well supported, User-Agent less so and for very good reasons (there's too much variation in UA strings)
> AMD automatically strips these headers out of requests to support caching for faster delivery.
> I need the Vary HTTP headers: AMD can cache the associated object if the Vary HTTP header contains only "Accept-Encoding" and "Gzip" is present in the Content-Encoding header
(AMD in this case standing for Akamai Media Delivery)
Short TTL makes you independent from hosting third parties, in that you can quickly change which hosting provider your domain name points to.
You can't win this one by only changing your TTL. The best solution is to use short TTLs and multiple nameservers on different providers.
If one dept deletes a record and the other doesn’t, how do you decide who’s right?
You could add a third dept that gives them both orders, but now that third dept is a single point of failure.
Though, I suppose if I'm responsible for it, I fix it faster for myself.
You want a protocol that gives consistent "global" state without any centralized / trusted users - blockchain/bitcoin is one of the only technical solutions to provide that.
I agree that it's a garbage solution in practice, but that's why it's got cryptoshit bundled in.
A potential different solution to DNS monopoly, if that is a problem that needs solving, is multiple name-resolution providers that have differing records on what name points where. (The tradeoff is that an owner may need to register their name with multiple different providers).
The world you describe, effectively with multiple roots, is coming. Russia have a switch (they’ve even tested it), to anycast out the root DNS IPs within the country, and block them externally. In theory this doesn’t make another “internet” (if IP space is still globally routable,) but in practice it does. Don’t be surprised if other countries follow suit (should they fail to leverage control of current infra via ITU or something.)
Of course that requires the server to properly fail, i.e. stop responding to requests. That doesn't seem to be the case here
https://github.com/octodns/octodns
DNS is fastest first* rather than main/failover. If AWS DNS was down your GCP DNS would have replied (if all is well) sooner than {timeout} so your visitor would still have a response
* Sort of. I think if the client doesn't get a reply from the server it picked randomly in 1s they move on to the next server, repeat until all fail
(I don't know if this is how it works, but I thibk that's how it supposed to work)
$ dig NS service.gov.uk +short
ns-cloud-e4.googledomains.com.
ns-cloud-e3.googledomains.com.
ns-cloud-e2.googledomains.com.
ns-cloud-e1.googledomains.com.
ns-831.awsdns-39.net.
ns-1983.awsdns-55.co.uk.
ns-117.awsdns-14.com.
ns-1080.awsdns-07.org. whois ja.net
Domain Name: JA.NET
Registry Domain ID: 499794_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.demys.com
Registrar URL: http://www.demys.com
whois uu.net
Domain Name: UU.NET
Registry Domain ID: 5486163_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Is it possible to see if/where is gov.uk using GCP or AWS for its domain zones? From what I can see -- that's not the case? Or am I looking in the wrong place?
Multiple vaults too is nice but I know others have ways to limit exposure of passwords in similar manners.
Bitwarden has 2FA support too, but does not have the unique key feature that 1Password has.