February 28th DDoS Incident Report(githubengineering.com) |
February 28th DDoS Incident Report(githubengineering.com) |
re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.
"Hey Jim, Did you go adjust that thing?"
vs.
"10:15am [JIM] /chatbot adjust x to y"
In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very least, configured to listen for and respond exclusively to connections from known front-end or application services.
Is this sort of deployment architecture falling out of favor? My casual observation is that cloud architectures—at least the ones I've seen employed by small organizations—are more comfortable than I am with services running with public IPs. What is going on? Am I misunderstanding this in some way?
When it's easier to just open up a server to the wide world than it is to learn how to connect safely, you'll always get a lot of people doing it.
(I’ll be here all week)
Those are the excuses I dealt with when I took over the current IT department. By now, only haproxy accepts public connections. Everything else is firewalled to the office at most.
Combine this with staying on top of vulnerabilities, this is really all you can hope for from a host standpoint. What is changing are the days of perimeter defense. The Zero Trust model is really the best path forward, and the only way to implement security in relation to the IoT.[1][2]
[1]https://www.youtube.com/watch?v=k80jOH2H10U [2]https://www.safaribooksonline.com/library/view/zero-trust-ne...
But when I read that he had found a public facing Jenkins server owned by Google, I figured I must be missing something.
I run a 2 man shop, but I still keep things like Jenkins behind OpenVPN. Why would anyone leave Jenkins open? There must be a reason, right?
https://emtunc.org/blog/01/2018/research-misconfigured-jenki... [0]
To answer your second question, I work for an open source non-profit software company, and we run some of our jenkins servers, which do continuous integration builds, publicly available so that community contributors and users can see build failures. Google has a number of open source projects that probably have similar goals.
A quick Shodan search[1] shows like 90k boxes publicly accessible Memcached. Misconfiguration of firewalls is a serious problem.
I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the source IP of another memcached server and let them burn each other out...
https://github.com/memcached/memcached/commit/dbb7a8af90054b...
How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!"
Naive decentralization (naive trust) doesn't work.
1. It may be difficult/expensive to arrange for the correct set of source subnets to be available at the points where filtering needs to be done. Motivation to perform egress filtering fails to overcome this cost threshold.
2. Fear that some customers are actually (probably without realizing) relying on alien source address traffic being routed. Therefore filtering that traffic would result in unhappy customers and support workload.
In our network over the years I've come across several instances where it turned out we were (erroneously) relying on one of our upstream providers routing traffic with source IP from another provider's network. Since policy-based source IP selection on outbound traffic is quite tricky to setup and get right, I can imagine that ISPs would take the easy way out and just pass the traffic.
https://www.internetsociety.org/blog/2014/07/anti-spoofing-b...
If I understand the article's point, essentially, carriers pay for the egress traffic that causes DDoSes, that cost and the cost of the generated ill-will outweighs that of filtering, whose price has fallen and continues to fall.
Personally, I think that if the article author is correct, then I wonder if this is one of those high-level long-term decisions that companies appear absolutely incapable of making. (In my experience, short-term gains are way overvalued at the cost of long-term loss, generally, especially when it is hard to directly determine the costs/benefits involved.)
It's still possible to restrict it, but simple RPF checks don't always cut it.
There is almost no reason whatsoever for clients to spoof their public IP address. Obviously, there are reasons to SNAT at the carrier level for load balance or routing purposes.
- Volumetric attacks like this one, mostly reflection.
- Application level attacks like SYN floods or protocol-specific attacks.
Defending against both costs a LOT of money.
Volumetric attack are dealt with at the network edge using rate limits and router ACLs. They're really easy to identify and block, but the point is that you need more bandwidth than the attacker in order to successfully do so. With attacks in the terabits-per-second range, this gets expensive.
Application-level attacks are harder to execute since there's no amplification and you need more bandwidth to pull it off, but they're much harder to block, too. They exhaust the server software's capacity by mimicking a real client. Common examples are SYN or HTTP floods.
When you get hit by a DDoS attack, you have two choices:
- Filter the attack and block the offending traffic without affecting legitimate requests. This is hard, and most companies can't do this. They need to have someone like Akamai on the retainer and dynamically reroute traffic like GitHub did.
- Declare bankruptcy and announce a blackhole route to your upstream providers (taking down the host in question, but protecting the rest of your network).
When you host custom applications that can't be scaled out or cached, DDoS mitigation is especially hard since you cannot just throw more servers at it like CloudFlare does.
Most services we host use proprietary binary UDP protocols, which is unfortunate, since UDP is easy to spoof and even experienced DDoS mitigation companies have trouble filtering it. Our customers get hit by DDoS attacks 24/7, so blackholing is not an option.
We had to build our own line-rate filtering appliances in order to handle the ever-increasing number of application-level DDoS attacks, by reverse engineering the binary protocols and building custom filtering and flow tracking.
All of this costs a huge amount of money, and most ISPs simply lack the resources to do this.
Happy to answer questions, but I'm going home right now, so it may take a few hours :-)
(Nitrado is a leading hosting provider specializing on online gaming, both for businesses/studios and regular customers, so we're dealing with DDoS attacks on a regular basis. We got hit with the same memcached attacks than GitHub and CloudFlare, and it was the largest attack in our company history. Ping me if you want to talk.)
Based on how it's done, you can't check first if the page hidden behind clouflare is something you'd want to enable javascript for, because clouflare will not let you see the HTML code of the page, without enabling javascript for it first.
That is broken.
We make things more annoying for VPN traffic because it's 99% bad actors. Every time someone is up to no good on our services, they're behind Tor/VPN.
It's simple cost/benefit analysis. If you think a business should bend to every single whim someone might have, then you haven't built much of one.
Making someone run Javascript so they can click on a captcha? Worth the loss of a few pennies because someone's angry about it on HN.
You need to conveniently ignore why people use Cloudflare to say that Cloudflare is breaking the internet. Ideally, nobody would have to use it, but that isn't reality.
Does anyone have an example of this webpage?
Unless I am engaging in e-commerce, I do not run a browser JavaScript engine. I rarely if ever encounter a webpage that truly "requires" one. GitHub certainly does not require JavaScript for me to use it via www.
Edit: The attacker didn't need nearly that kind of bandwidth to execute this attack. See [1]
Edit: 1/50th -> 1/400th (bits vs bytes)
[0] http://www.internetlivestats.com/one-second/#traffic-band
> The vulnerability via misconfiguration described in the post is somewhat unique amongst that class of attacks because the amplification factor is up to 51,000, meaning that for each byte sent by the attacker, up to 51KB is sent toward the target.
From what I understand the attack originates from publicly exposed memcached servers configured to support udp and that have no authentication requirements:
- put a large object in a key
- construct a memcached "get" request for that key
- forge the IP address of the udp request to point to that of the target/victim server
- memcached sends the large object to the target/victim
Multiply times thousands of exposed memcached servers.
That about right?
I consider it community service.
Cloudflare provides some page with JS code that computes something and then procedes to the correct page if it is computed correctly.
So I can either enable JS both for the cloudflare interstitial page and for the target website, or I'm not able to access the website.
I'm not angry, I just close the website/go back to search results. I will not allow the browser to run random JS code from the target website for no reason, just because the interstitial page requires it.
Still, if someone requires computing random challenges in javscript in order to gain access to a web page, they're breaking the web. Javascript is still an optional addon.
And it doesn't cost any significant amount of money except initial configuration and automation. The "CPU power" to add an ACL on interfaces is negligible.
Google 'DRDoS attacks' to learn more. They are responsible for most of the largest volume attacks IIUC.
It strikes me that this would prevent a massive number of amplification attacks.
There are still a zillion low-end bottom feeding web hosts who wouldn't do anything about this, though.
This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before.
Edit: It looks like Github uses NTT for traffic. Hello Github Netops person, you need to call your sales rep and turn on DPS Lite. It's like $100 per 10gig port and you get full ACLs. Telia, another one of your carriers, will do this too. At least they have for me. Level3 though? Lol kick that sorry network to the curb
Also, get another /22 allocation so you can at least separate out your DC-origin traffic from your customer traffic.
I mean someone can be right, but that doesn't give that person the right to be dismissive. Treating ignorance with disdain isn't going to make anyone smarter.
It's a requirement when a CloudFlare'd site is in "I'm under attack" mode.
"What's also cool is that data on attack traffic that doesn't pass the automatic checks is fed back into CloudFlare's system to further enhance our traditional protections."
"[P]re-vetted traffic"?
Does this mean they are whitelisting certain IP addresses?
GoogleBot can make hundreds of requests and double digit parallel connections, as frequently as they like, but a single user making one request and one connection is blocked because they are not enabling Javascript?
This does not sound like an intelligent filter.
"[K]nown legitimate visitors"?
What exactly does this mean? How do they "know" a visitor is "legitimate"?
"[A]ttack traffic that doesn't pass the automatic checks..."
Is it possible that non-attack traffic could fail the checks?
What about a single request from a single IP that does not pass the checks because the user does not have JavaScript enabled?
Does the IP address end up on some blacklist?
I have seen Cloudflare reject connections based on certain user agent strings, a header that everyone knows is user-configurable, arbitrary and not a reliable indicator of anything meaningful.
This despite volumes of "legitimate" traffic from same source preceding it. Pick wrong user agent string and suddenly the source becomes "illegitimate".
It would be interesting to know what "checks" the Javascript in question is performing.
Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!
If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service).
Does this prevent people from making poor decisions, like just blindly starting the service without reading the configuration file, or disabling firewalld/enabling a rule without checking the configuration first? No, it doesn't - but that small hurdle at least prevents people from inadvertently turning on a service and opening it up to the world just by installing a package.
This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.
Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain.
Security is more important than developer velocity, but users pay the bills.
> The memcache protocol was never meant to be exposed to the Internet, but there are currently more than 50,000 known vulnerable systems exposed at the time of this writing. By default, memcached listens on localhost on TCP and UDP port 11211 on most versions of Linux, but in some distributions it is configured to listen to this port on all interfaces by default.
Yikes!
That is incorrect.
The attackers made requests that were forged to have the sender IP address of Github to multiple public memcached instances. Memcached then responds back to Github instead of the attacker.
This is documented in more detail in the Cloudflare vulnerability report[0]
https://blog.cloudflare.com/memcrashed-major-amplification-a...
It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?
The problem is the ISPs allowing spoofed IPs.
So the attacker only needs to find somewhere on the internet that is capable of generating spoofed packets. They needed a lot of places that had a reflection server, but the requirements for the spoofing was much smaller.
In other words, you would have to prevent 99.9% of the internet from being able to spoof source addresses before you fixed this problem.