My cat alerted me to a DDoS attack(dannyguo.com) |
My cat alerted me to a DDoS attack(dannyguo.com) |
I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.
No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.
In the recent NYC ones there are videos of dogs howling before any of the tremors are noticeable by people. This is a common phenomena I believe.
There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.
See https://manoa.hawaii.edu/exploringourfluidearth/physical/oce... to verify that there are two types of waves, and the P waves arrive first.
At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.
Somewhere deep in my brain, there are neurons that developed for some more evolutionarily-relevant purpose and which are now a little disgusted with how they’re being used.
I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.
Not replying is the only valid answer. Trolling them could potentially put you more on their radar and get targeted for other attacks. And for what?
One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.
But also, on a tangent, there is a bird that does this kind of non-conspecific alarm calling the time as part of its food-gathering strategy: the African fork-tailed drongo.
The drongo gives true alarm calls to food-rival species nearby, to tell them when it has spotted a mutual predator. This leads to these food-rival species coming to rely on these signals. But then, every once in a while, it gives a false alarm, to get the food-rivals to run away for a bit, so it can nab the bugs/berries/etc that the rival would have been eating.
Don't want to spoil. It's nice and short and a must-read for cat lovers. "The Price".
An inverted Schrödinger cat.
Ah, the days before ChatGPT!
On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?
While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.
Regardless, very cute - what’s your cat’s name?
Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.
“Sir you need to leave that mutt outside!”
“He’s a service dog”
“Why? You don’t look like you have any disabilities”
“Wow. First of all – rude! Second, yeah you are right I don’t but you see he’s my DDoS dog and I need him with me at all times to protect the company servers”
I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.
You could still overload the service with a sufficiently large attack in either volume of connection requests or number of unique IP addresses.
Token buckets are usually part of an overall resilience strategy rather than a silver bullet to solve all denial of service concerns.
For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?
I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.
That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.
1. We didn't experience that many incidents that couldn't wait until working hours.
2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.
It stops working when the company grows and no one understands the whole system and you need on-calls from several teams. Then the company does some formal on call rotation and it's fine again. It hurts during the transition only.
It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues
Also, if you can get an equivalent role with less requirements such as being on call, then I guess it is just a question of grabbing it!
But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.
Anyway, better experience than being woken up by a dozen SMS alerts.
... that you know of
Talks about useful tech
"iS tHiS An aStroTurFing Ad"
It does +1 most of those types of spam farms with a bit more technical discussion, but not really that much.
cattackstrophic!
I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.
When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.
Since using LCD monitors or laptops, those symptoms have disappeared.
When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.
Thus,
Even though there was no sound, wouldn't your cat have sensed that as well?
1. $50k or we attack - didn’t register anything
2. $25k or else - a minor overload on the server but nothing serious.
3. $10k or else - a serious attack which affected the service in a major way.
4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.
We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.
The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.
From the german chaos computer clubs yearly meeting. Linus talks about what to do and who ransoms work, how "well" the service is and briefly pros and cons of paying.
https://media.ccc.de/v/37c3-12134-hirne_hacken_hackback_edit...
Also a good one was the first part: https://media.ccc.de/v/36c3-11175-hirne_hacken
To puff and look important and to say
Though we know we should defeat you
We have not the time to beat you
We will therefore give you cash to go away
And that is called paying the Dane-Geld
And we've proved it again and again
That if once you have paid him the Dane-Geld
You never get rid of the Dane
Somewhere I read that some ransomware had excellent "customer" service for helping you transfer over the payment and promptly restore your files.
If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.
Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.
That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.
It would be pretty cool if there was a way to DDOS-harden at the protocol layer. Not sure if that’s even possible though
And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.
Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.
Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.
Oh, this poor guy is being DDoS'd, so we're going to make sure that their service remains denied.
Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.
With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
> Ah, the days before ChatGPT!
The topic made me read that as CatGPT, and now I can't pull it out of my head.
If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.
You can somehow subconsciously detect those in your SO without doing it on purpose.
I wouldn't call this a permanent on-call, just being responsive.
If you're running on a single IP, yes. If you're running on multiple IPs, it's not that bad for the one that's being attacked to get its traffic dropped and everything else works. It's not great, but what are you going to do. If you've got enough traffic to overwhelm the inbound on the top of rack switch your box is on, you're not going to be able to really serve any of the good traffic anyway.
> With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
There's usually way too many source addresses to do that, and anyway, routing infrastructure is geared towards looking at destination addresses, not source addresses. Also, each individual source doesn't look that bad --- if I've got 10,000 sources each sending me 1 Mbps of garbage, nobody is going to accept a block for only 1 mbps of sending, and yet, there's 10 Gbps of garbage arriving at my box; if I've got 10 Gbps or better connectivity, no big deal. But, if I'm only on 1 Gbps, I'm getting less than 1 in 10 of my inbound packets. I'd argue, if everything else has a big enough connection, it's probably still no big deal, it should be able to drop packets headed to me, as long as its upstream connection isn't filling up. But once abuse is causing contention that impacts others on my rack, it's probably time to null route.
If it's one of the big botnets with 100,000+ compromised systems, the individual bandwidth is even less. And if the botnet has significant ability to deliver spoofed traffic, source based filtering is meaningless. If it's reflected DDoS, I dunno --- there's value in hunting down the chargen services and removing them from the internet, but that's usually a lot more work.
OTOH, look on the bright side, if your outbound bandwidth is high and you get a lot of inbound DDoS, you may have roughly balanced your usage, and you may qualify for settlement free peering! (IMHO, this has got to be a major part of Cloudflare's business plan)
An ISP has to do what it has to do to save their business when a large attack hurts their business. This may be what the attacker wants but that's not an excuse to do nothing.
But what machine learning has to do with this is not clear. Null routing protects you against traffic volume. Machine learning sounds like it maybe can help diagnose more sophisticated low-volume attacks. Maybe. You don't want anything remotely compute intensive when mitigating attacks.
In fact, there is a family of such birds, the honeywarblers, who locate beehives, then find humans and lead them to the beehives.
Wikipedia says that the behavior is dying out because there aren't enough human foragers.
From their side, the logic seems as simple as: repeat the attack -> some chance of more money, don't repeat the attack -> 0% chance of more money