DNSimple target of DDoS attack

DNSimple target of DDoS attack(twitter.com)

64 points by markprovan 12 years ago | 28 comments

the9to5 12 years ago |

Bravo to the folks at DNSimple for being on top of communications during this time. It's something that they themselves brought up less then a year ago during the Zerigo DDOS (https://news.ycombinator.com/item?id=4280515) so it'd good to see them sticking to it.

But it also seems as though the same advice proposed in that thread should have been used by their customers: Namely, utilize multiple DNS providers to mitigate risk, and choose providers with IP anycast. Heck, even setting up your own secondary DNS on a $5/mo cloud server would keep your site up (unless of course your site is the main target of the DDOS).

whafro 12 years ago |

It's not a great long-term solution, but DNSimple has been disabling their ALIAS support, which many Heroku and AWS users are likely to depend on, especially if you use SSL.

Short term, keep your ALIAS record and add an additional A record for your root domain pointing to one of the IPs indicated by your hostname. DNSimple says they'll treat the A record as a fallback when ALIAS isn't working, and will return both sets of records when it is (https://twitter.com/dnsimple/status/341574753276002304).

For the next 3/12/24/96 hours or however long it takes for the threat to subside, this should increase your availability, and the likelihood that your A record will work for that time is probably reasonable. Longer term, you'll want to get rid of the A record.

yesimahuman 12 years ago |

Ah, got bitten by this. Just added Route 53 as redundancy, should have done that a long time ago.

whafro 12 years ago | |

Are you doing this manually? I make solid use of DNSimple's ALIAS record type, which is also supported by Route 53, but DNSimple says they don't support AXFRs (http://support.dnsimple.com/articles/master-slave-support).

yesimahuman 12 years ago | | |

Honestly, I'm not a DNS pro, but I added more nameservers in our registrar to point to R53 nameservers, then setup R53 to point to our Heroku app w/ alias support by following this guide: https://devcenter.heroku.com/articles/route-53

nate 12 years ago | |

Are you using route53 to do an alias AND have the root domain use SSL? The setup at route53 is confounding me. I thought I'd setup a CNAME www.domain.com that points to sub.herokussl.com, and than an A record Alias that points domain.com to www.domain.com. But Amazon seems to not avow that the CNAME record is a "Record Set" that I can use for this.

whafro 12 years ago | | |

That's interesting. Have you tried it the other way around?

- domain.com ALIAS sub.herokussl.com - www.domain.com CNAME domain.com

yesimahuman 12 years ago | | |

Did you set up the S3 bucket with static file hosting redirecting to the www subdomain, before you try adding the alias in 53?

soci 12 years ago |

DNSimple seems the best way to go if you want to host your service with Heroku using a root domain (no www at the beginning of the domain name) [1].

Unfortunately, DNSimple is now the weakest layer of our stack. And at http://KiteBit we are suffering it right now!

[1]https://devcenter.heroku.com/articles/custom-domains#root-do...)

robotmay 12 years ago | |

Yep, I just got whacked by this at https://www.photographer.io (www is working at least). Guess I've learnt my lesson about using the root domain.

However I'm glad this happened whilst I was still beta testing! My CloudFront stack was pointing at the root domain, which was stupid. Fixed that now.

soci 12 years ago | | |

We were not so lucky.

We have a URL forwarding set to www that points to out root domain so we are completely down.

jwarzech 12 years ago |

Our site (and others) seem to still be working through the 'www' domain. We have had nothing but great experiences with DNSimple up to this point and will probably stay customers, just sort of frustrating as we wait for our domain to resolve to another dns provider as a quick fix...

robotmay 12 years ago | |

I'm having an interesting issue where my root domain is currently getting its nameserver details from DNSimple, whereas www. is getting them from iWantMyName. Hopefully that rotates around too at some point soon :\

zrail 12 years ago |

Is anyone else seeing a sustained level of trashy DNS queries to their own servers? I've been seeing a sustained level to mine that's way above normal, for the last few days. I wonder if this is a broader problem than just DNSimple.

devicenull 12 years ago | |

That's probably someone attempting to you for a DDOS reflection. Take a look at http://openresolverproject.org/ and make sure you're not providing an open resolver to the internet

zrail 12 years ago | | |

Definitely not in the list for either server.

dsl 12 years ago | |

Can you give an example of what 'trashy' is?

You could be an open resolver being used for reflection. You could be running DNSSEC and providing an amplification vector. You could be getting queries for another DNS server that used to have your IP. The possibilities are endless.

zrail 12 years ago | | |

I'm running tinydns so it's definitely not DNSSEC or open resolving. The queries are for my domains, so it's not that they're questionable. The problem is that there are millions of them all from the same IP. So, it could be that it's some kind of attack, but I really don't know.

I'm planning on moving hosting over to Route53 at some point in the next few days, because the fact that I can't figure this out other than "turn off logging" tells me I probably shouldn't be running DNS servers.

thejosh 12 years ago |

This seems to happen quite often to DNS hosts, I remember ClouDNS getting hit often.

It's easier to hit these sorts of "smaller player DNS hosts" if the website you want to take down is otherwise protected?

relix 12 years ago | |

In the three years I've been with DNSMadeEasy [0], haven't had an outage yet.

[0] https://cp.dnsmadeeasy.com/u/62796 (affiliate link)

randall 12 years ago |

Anyone know how to make DNS redundant? Is it as simple as adding them as extra nameservers, and then copying all the records? I'm thinking about using Linode's DNS as a failover.

colmmacc 12 years ago | |

There are two important steps;

1. Adding NS records to the parent zone via your registrar. E.g. if you are using example.com, when you add nameservers with your registrar they add them to the ".com" zone.

2. Update the NS records in your own copies of the zone on your DNS providers.

If your registrar is also one of your DNS providers, then both of these steps are sometimes handled in one action from your registrar - but you still need to update the NS records on the other provider.

"NS" record sets are special in DNS in that there is a copy of the NS record for a particular zone in both the parent zone and the child zone. About 8% of resolvers consider the parent zone's copy the one that matters, the other 92% honour whatever is in the child zone's.

This can lead to confusing cases where you have different NS configurations on different providers - the resolver may "stick" to whichever one it found first (as long as both providers are in the parent zone). DNS can be maddening!

Full-disclosure: I'm a Route 53 developer.

robotmay 12 years ago | | |

This is unfortunately a bit tricky as DNSimple doesn't let you update your NS records for the root domain with them. My computer seems to be going through DNSimple still, but Pingdom went via iWantMyName to find my new Route 53 nameservers. Not sure what I can do for now though.

latch 12 years ago | |

Yes. You can do master-slave with AXFR but it doesn't seem very well supported nowadays. I remember 1 provider that let you put in your AWS credentials and they'd sync to Route53 using the API, though I can't find which one right now. Different providers offer different solutions...a lot don't offer anything.

Of course, you can always do it manually..which is fine if you have few records and they are static.

edit: Kept googling and I did find http://www.dnsly.net/ neat enough that I wanted to share it, even though I think dns providers should be doing this as part of their existing packages.

Also, although it's no guarantee, dns providers that use anycast are less susceptible (but not invulnerable) to ddos.

dexcs 12 years ago |

That makes sense. rubygems.org was down for me for a few minutes....

davidradcliffe 12 years ago | |

We're working on adding some redundancy!