Ask HN: How did Dyn fail to fend off DDOS? I'd imagine that DDoS attacks is something that DYN and other DNS providers would spend a lot of resources to prevent. Was there something specific about this DDoS attack that DYN was unprepared for? Or is there some reason that distributed natural of DNS makes it hard to prevent DDoS? Anyone know of any steps that DNS guys are taking to prevent another DDoS? |
A serious conversation with vendors about default passwords and backdoors post this incident will help prevent recurrence. This has forced this talk and we are better for it.
There was a time when your windows box would get popped from being online for more than 4 minutes. We recovered from this. Conficker in 2008. Blaster in 2003. It was a 'BIG BOTNETS OH NO', but we cleaned up, recovered, hardened. Microsoft went from being botnet enabler to an active force in dismantling bots and crime rings. It sucks, and some of us have a bad day, but we recover ever stronger.
XiongMai Technologies may well find themselves in some international hot water over this incident, and I think they deserve it. They sold a faulty product that caused billions of dollars in lost revenue to some very large internet properties for a day in October 2016. I would encourage vendors look at these incidents from last decade and how these were turning points for upping their security game. I would encourage its victims to investigate legal recourse.
Specifically the current vulnerable nodes of Mirai, i am sure these will be removed from the internet pretty soon. One only gets to fire something like this a few times before the feds are on the door.
Your regularly scheduled program will commence shortly.
These devices need to have an update mechanism. The manufacturer needs to have an ongoing security effort, across their whole device line (probably a significant investment in development resources and process -- consider that right now, the firmware for a device is probably coming off of a firmware dev's laptop; I've seen this happen at a big company). And devices will have to be sunset, to control the ongoing cost. Consumers will love that.
I don't think we're doomed, exactly, but it's probably always going to be a problem. And there's probably a market for embedded firmware application layers that don't suck, for starters.
It's all well and good saying that, and yes, if manufacturers are repeatedly/grossly negligent then maybe they should pay compensation and/or punitive financial penalties. However, unless you know something the rest of us don't about how to guarantee Internet-connected devices are perfectly secure, that sort of financial pressure can't be the whole solution, or even the main part of the solution. Ultimately, it may just mean that smaller players can't afford to risk participating in the industry any more, and no-one will be better off if reduced competition is the main result of this. We must be able to handle this more constructively than just demanding perfection and punishing those who inevitably fail to deliver it.
It's been said before but I will repeat it here; manufacturers have no reason to expend any resources on security until they are held liable for the damage they facilitate. We must make selling insecure devices a liability just like selling unsafe devices is in meatspace.
There are a few other ways to handle the problem of securing endpoint devices. For example, for devices that are intended to use a local aggregator, gateway, or proxy of some sort you can get around the issue (and improve the UX) by avoiding passwords entirely, and requiring that the device instead be paired with a base station through a physical action the user performs (pressing a button on both, knocking them together, etc.) instead.
We also need some big recalls. If Homeland Security tells the Consumer Product Safety Commission this is a national safety issue, the CPSC can order a recall. Something like this worked with those exploding "hoverboards". CPSC ordered recalls, Amazon took the junk back, and Amazon refused to pay manufactures in Shentzen. The manufacturers were furious, but hoverboards with crap batteries disappeared from the market very fast.
The unfortunate truth is that with the Internet of Things the amount of devices that can easily be taken over has grown so fast that we see DDoS attacks of unprecedented size. Even more unfortunate is that there is no sign whatsoever that this is going down again.
Not quite, the "IoT" botnets are particularly small in the great scheme of things. Google "conficker" for example.
Edit: Interesting how this is getting downvoted so much. Conficker had up to 15 million nodes, far bigger than any "IoT" net (when did home routers become IoT anyway?). It's far easier to build such huge windows nets because you get millions of insecure computers with relatively standard hardware and software, not so much with "IoT".
In the past decently sized botnets simply weren't used to send DDoS attacks as much, that's all that's changed.
In order to defend against a DDoS attack, you really only have two options. One is to have sufficient capacity to cope with the extra load without undermining your normal service. The other is to reduce the amount of extra load you have to handle, by identifying and blocking the hostile traffic at some point before your main system deals with it fully.
In this case, the scale of the attack was huge thanks to all the woefully insecure IoT devices out there. But worse, from the initial reports it appears that the requests being sent were effectively indistinguishable from valid DNS requests: they came from diverse sources, and asked DynDNS to do exactly what it's normally supposed to do, just for random subdomains that don't actually exist. Unless there is some pattern in those requests that allows for identification of the hostile incoming traffic so it can be dropped early, there's probably very little DynDNS could have done here. And of course the attack is particularly effective because by taking out infrastructure rather than attacking a specific site, it brings down large numbers of high profile sites all at once.
It is disturbing, but apparently the reality we face, that there are now so many hopelessly insecure devices on the public Internet that this is possible. The best long term strategy for dealing with it seems to be trying to improve the standards of Internet-connected devices and reduce the number of highly vulnerable devices with access to the Internet, but this was always going to be difficult with IoT products aimed at the general public. I suspect some sort of remediation/recall scheme for manufacturers/vendors and some sort of throttling of users' Internet connections to force them to respond to security recall/update notices may be necessary if this kind of attack starts to become a pattern.
NANOG 68 BackConnects Suspicious BGP Hijacks is shown 4ish days ago. Last talk of the night, discusses BGP hijacking shenanigans and krebs; touches on MO of possible attacker. Speaker is Director at Dyn. Attack in retaliation.
So far the targets have been organisations that have responded to or made allegations of corrupt DDoS business.
Please don't buy into all this cyberwar bullshit, this may just be a well resourced (its really not that hard to pop boxes with default passwords.....) attacker doing criminal response to commentary.
There's also no small amounts of publicly available evidence that Backconnect used insider information provided by their CEO (ex Staminus employee) to compromise Staminus network earlier this year by hijacking a management range of theirs.
recently there has been an aggressive uptick of dns ddos attacks against smaller companies/service providers that run their own dns infrastructure. this includes small/regional internet service providers and individual sites/hosts that still run their own servers.
in almost all of these cases that i'm aware of, the smaller companies immediately outsourced their dns services to a larger company, one that ostensibly is able to either absorb, scrub, or otherwise defend against these types of attacks.
extrapolating to a global scale, what's happening is a forced consolidation of dns infrastructure into a handful of large players. even in the case of having redundant providers, it's usually two very large providers. and as we just saw today, a terabit-level attack is not something we can readily defend against. what if there's even more in reserve?
in other words, we're putting all of our eggs into one basket. and someone is aggregating enough attack capacity to take out nearly the entire internet at once. it doesn't help that everyone is voluntarily consolidating their infrastructure onto a small handful of public cloud providers.
we are setting ourselves up for a massive internet outage.
In a DNS based amplification attack, you use several DNS servers to take down some other unrelated service, this time it's just a lot of devices in a botnet attacking the DNS servers directly.
1. https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns...
2. https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with...
3. https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...
So not quite.
> Dale Drew, chief security officer at Level 3, an internet service provider, found evidence that roughly 10 percent of all devices co-opted by Mirai were being used to attack Dyn’s servers. Just one week ago, Level 3 found that 493,000 devices had been infected with Mirai malware, nearly double the number infected last month.
http://www.nytimes.com/2016/10/22/business/internet-problems...
If they aren't significantly underestimating the number of devices participating in this attack, it paints an ugly picture of things to come. My understanding is these botnets are almost impossible to eradicate due to how fast/easy it is to re-compromise the devices, so traditional methods of taking out C2s do almost nothing. Bonus - Mirai source code is freely and easily available for skids to use now, so there's no single threat actor for attribution/retaliation/arrest/etc.
https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/
http://www.csoonline.com/article/3123797/security/some-thoug...
The problem will continue, and may get even worse, since many of the insecure internet attached video cameras are insecure because of passwords hard-coded into the devices; they can't be easily made more secure.
If I'm not wrong, it's only preventable by increasing the resources of the server, doing anti-bots things like CAPTCHAS (not feasible for stand-alone IoT devices) or detecting weird patterns (which can be masked really easily).
How will DDoS attack be preventable in the future? There will be so many things and nano-thing connected to the internet that can act as "attackers". Is getting harder and harder everyday.
https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...
I don't know any other independent researcher who confirms this.
Is it that simple? or am I missing something?
The vigilante hackers is for for comic books IMHO. I would trust a 3-letter gov org. Maybe the NSA would be a lot more useful if instead of breaking the internet, trying to fix it.
IIRC, US consumer law requires the consumer to be the victim. (IAAL/NY, but not practicing) This restriction is called privity – the exceptions to privity are narrow, and no exception comes to mind here.
In this case the primary victims, the online services, are third parties, with any consumer recourse blocked by privity.
These third parties arguably have a couple options, though. The first and perhaps most theoretically interesting is the "class defence", the procedural complement of a "class action", where a few people (the third party online services) can sue multitudes (owner-operators responsible for malicious devices on the Internet) in a single process. Were such a case brought forward, these consumers could sue the manufacturers for indemnity. While as a litigator this makes the most theoretical sense, and this procedure exists in at least one jurisdiction I know of, I have never seen it tested.
Arguably a better option would be for the third parties to sue the manufacturers for negligence, based on the obligation that the manufacturers have to the public.
Any litigation is fraught with uncertainty though, not least of which is having a member of the judicial bench who is capable of properly evaluating the facts (which is not to say they are not out there, but they remain rare).
Like most externalized costs, the recourses of affected individuals are slim and ineffective.
> If Homeland Security tells the Consumer Product Safety Commission this is a national safety issue, the CPSC can order a recall
Proper regulation is a better choice, IMHO, though I don't know what the best process might be.
Wait why is Amazon responsible? Why should they be sued?
The only thing that will make a dent at the problem quickly, is wholesale filtering of all Internet traffic by all network providers originating from the IP addresses identified for being part of these botnets.
The original Mirai program tried a little over 60 passwords and it would just brute force into an IoT device.[1]
From what I read, it seems that one specific manufacturer in China is the owner of a lot of devices used in the Mirai botnet attacks.[2]
1: https://github.com/jgamblin/Mirai-Source-Code/blob/master/mi... 2: (I cannot find the link, but it was an article from yesterday)
EDIT:
Found this when googling the strange '7ujMko0admin' password in Mirai: http://www.cam-it.org/index.php?topic=9396.0 So it looks like the Chinese manufacturer that they target is Dahua.
I believe the Dyn attack was via Mirai also.
Ex. "I can't believe our network can't handle that traffic, it is only 20 milliKrebs!"
Although the source code is out there, those will not be able to control all those devices.
“The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present.
- https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...
That's not to say it couldn't flash the devices but I don't recall seeing that capability in the Mirai source and haven't read about it doing so.