How Stack Overflow plans to survive the next DNS attack

How Stack Overflow plans to survive the next DNS attack(blog.serverfault.com)

133 points by samhamilton 9 years ago | 40 comments

matt4077 9 years ago |

I think I measured Cloudflare's performance and chose it over Google because it was consistently faster. If the stack-stackers are reading, I'd love to hear why they didn't make the list.

Also, it'd be a great public service to publish the results. Even if it's just enabled for a day per year or so the results would probably appreciated by many. And you could always sell your altruism as the need to continually monitor the situation :)

thefarseeker 9 years ago | |

Author here. CloudFlare's DNS was measurably, and consistently, faster than almost every provider we tested. So I can concur with you in that regards. It was ultimately faster than Route 53 and Google Cloud (the path we ended up going).

However, when our relationship with CloudFlare ended and we moved to Fastly, one of the reasons we did so was unreliable lag with DNS updates at CloudFlare. Sometimes it would take minutes or hours for DNS changes to be affected in their system, and they could never provide a satisfactory reason why.

samhamilton 9 years ago | |

+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN.

Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare?

[1] http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...

dx034 9 years ago | | |

The fact that they abandoned Cloudflare only 6 months after this post means that they must've been pretty disappointed. I wonder if this is the same reason as for some of the other pages that went to Fastly from Cloudflare (e.g. Imgur).

deno 9 years ago | | |

tl;dr: Javascript/Google reCaptcha Paywall.

Source:

https://meta.stackoverflow.com/questions/323537/cloudflare-i...

fstd 9 years ago | | |

Can't believe they're running on Windows. Yikes.

captncraig 9 years ago | |

One major issue with their dns was unpredictable lag while changing records. On most occasions we can change a record and it would propigate immediately. In some rare cases, it could take fifteen minutes, and in a few instances up to eight hours before the change would show up on their nameservers.

Cloudflare was unable to diagnose this or give us any comfort that it would improve.

bks 9 years ago |

Umm, brilliant thank you for this.

I ended up with a Dyn / Route53 configuration. We used libcloud to sync everything together. We also added the exported zone to Cloudflare but did not enable it.

We had actually planned for this, but in no way did we ever come close to your in depth testing. The @ Azure issue - thank you for uncovering this for the rest of us.

tibu 9 years ago | |

Can you maybe share how you did the sync between them? Also there are already some tools pulling zone data from Dyn and adding it to Route53? Can you share why did you choose an own sync. (I'm planning to do the same and I'm interested in other's opinions)

captncraig 9 years ago | | |

We actually wrote a tool to manage this. We define our desired records in a common dsl format, and the tool can interact with various providers to ensure things match the expected state.

We should be open sourcing this rather shortly, so stay tuned.

Sorry, I'm not who you asked, but that is how we are doing it at stack overflow now.

matt4077 9 years ago |

The calculation regarding the ideal number of name servers to list needs some empirical data regarding the likelihood of provider and server outages and the client reactions to it, right? Because otherwise 2 would must be the best number, if I'm not mistaken (Chance of hitting the provider that's offline is always 0.5 on the first try, but the second try would be guaranteed to hit the other).

Here's the math for expected number of tries if half of the servers are offline. (It's a hypergeometric distribution but I couldn't find a closed formula)

E(2 server) = 1 * 1/2 + 2 * 1/2 = 1.5

E(4 server) = 1 * 2/4 + 2 * 2/4 * 2/3 + 3 * 2/4 * 1/3 = 1.67

E(8 server) = 1 * 4/8 + 2 * 4/8 * 4/7 + 3 * 4/8 * 3/7 * 4/6 + 4 * 4/8 * 3/7 * 2/6 *4/5 = 1.73

thefarseeker 9 years ago | |

Author here. One thing I didn't cover in the post was about how to have Google and AWS honour their SLA's is that you have to use _all 4_ of the nameservers they provide. Because we're not doing that (only using half), we have to balance the chance of an outage versus the impact of an outage.

You are correct in saying that more empirical data could be used here. We might even end up changing our minds. I'm not much of a numbers person but I might pass this onto some of the people in our company who love solving problems like this.

jlgaddis 9 years ago |

It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers.

Hurricane Electric supports this but most of the providers mentioned in this article do not.

ksec 9 years ago |

From my experience EdgeCast and DNSMadeEasy were consistently the fastest DNS. I guess both were dropped because of price when Google DNS and Route53 did the a similar job.

And as other have said, while Cloudflare may not be for everyone, their DNS is possibly the fastest. Not sure why SO decide to drop them.

*Some old Data http://www.dnsperf.com/

I also wonder on the performance of DNSimple. But they dont see to emphasis much on performance.

thefarseeker 9 years ago | |

So DNSMadeEasy made it all the way through the barrage of tests. I even wrote a library for their API so that we could integrate it into our DNS software (https://github.com/mhenderson-so/godnsmadeeasy), but at the end of the day their performance in certain regions was not good enough. In some countries they were measurably faster than R53, but in others they were measurably much slower.

EdgeCast were dropped due to pricing, and that there's talk of Verizon selling the EdgeCast services again.

DNSimple didn't make it to performance testing because they only had 5 POPs, as opposed to 20+ of other providers.

CloudFlare's DNS was consistently one of the fastest, you are correct about that. If you read my responses to other comments here, you'll find that we decided not use their DNS service because of some fairly pervasive API issues we had with it.

elktea 9 years ago |

Netflix have a tool for this as well https://github.com/Netflix/denominator

captncraig 9 years ago | |

We have a pretty cool tool for managing this at Stack. Changing from a single provider to two can literally be done by changing one line of config. We should be open sourcing it very soon, and a blog post on it sometime thereafter.

skuhn 9 years ago | |

Denominator is not actively developed: https://github.com/Netflix/denominator/issues/374

Last commit of substance was in Sept 2015.

majewsky 9 years ago | | |

I have not looked at this particular commit history, but I want to argue against the notion that no fresh commits must mean that a project is abandoned. Some stuff is just mostly finished after some point.

paulddraper 9 years ago | | |

Weird. I attended a lecture about it not too long ago.

I wonder what Netflix is doing instead.

Mojah 9 years ago |

I'm currently working on a tool [1] that can help with checking if all your different providers are 'in sync' and responding with the same answers. Setups like these are only to grow more common as people realise a single DNS provider is a SPOF of its own.

Very good analysis of SO and a smart move to roll this out _before_ a new DNS outage!

https://dnsspy.io

SteveNuts 9 years ago | |

I've thought about doing something like this, the biggest issue I found is feature parity between DNS providers.

If you could have a unified API that would create the records on multiple providers that would be money, it's just that you'd lose out on some things like Route 53 health checking, etc.

cuu508 9 years ago |

Is there a good writeup somewhere about setting up redundant NS records at the zone apex? Or, more generally, "DNS primer for busy developer" article?

skuhn 9 years ago | |

Once you have configured your zone with multiple providers, it's simply a matter of adding NS entries for each provider's authoritative servers to your registrar. The harder part is ensuring that the zones are kept in sync and that you don't rely on features (such as GSLB stuff or ALIAS records) that aren't available with all providers.

It's up to the client resolver to handle failover, so it's not perfect in terms of availability, but better than nothing.

For example:

  $ dig ns amazon.com
  amazon.com.		3599	IN	NS	ns4.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns1.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns3.p31.dynect.net.
  amazon.com.		3599	IN	NS	ns2.p31.dynect.net.
  amazon.com.		3599	IN	NS	pdns1.ultradns.net.
  amazon.com.		3599	IN	NS	pdns6.ultradns.co.uk.

(note that this is also TLD redundant, since there's a .co.uk included)

captncraig 9 years ago | | |

The tricky part is making sure the apex NS records are consistent across all authoritative nameservers. A surprising number of dns providers do not let you edit those.

vaara 9 years ago |

I wonder why there's no consideration of anycast servers.

thefarseeker 9 years ago | |

Author here. Google and AWS are anycasted services. We were not interested in running our own anycasted DNS due to the management overhead of doing so, and because the cost of anycasting our own services would be orders of magnitude more expensive than outsourcing that to an established DNS provider.

$ dig ns amazon.com amazon.com. 3599 IN NS ns4.p31.dynect.net. amazon.com. 3599 IN NS ns1.p31.dynect.net. amazon.com. 3599 IN NS ns3.p31.dynect.net. amazon.com. 3599 IN NS ns2.p31.dynect.net. amazon.com. 3599 IN NS pdns1.ultradns.net. amazon.com. 3599 IN NS pdns6.ultradns.co.uk.