The story around the Linode hack(straylig.ht) |
The story around the Linode hack(straylig.ht) |
HTP ("Hack The Planet") is a group that likes to break into things. Another (unnamed) group of people impersonated a third group of people ("ac1db1tch3z") and tried to cause trouble for HTP.
The impersonators located HTP by examining one of HTP's botnets (a collection of compromised computers that are used to launch things like denial of service attacks). Botnets have to receive instructions (e.g., targets to attack) from somewhere, so it's likely that the impersonators followed the path taken by commands to the botnet, and found the network(s) that HTP uses to organize themselves.
HTP realized this, and wanted to get back at the impersonators. They found out that the impersonators used an IRC channel (chat room) hosted on a network called SwiftIRC. If HTP could break into SwiftIRC (which is hosted on Linode), they could cause all sorts of trouble for the impersonators. So HTP decided to break into Linode, so they could break into SwiftIRC, so they could break into the group of impersonators.
To break into Linode, HTP broke into their domain name registar (name.com). They planned to secretly take control of linode.com, and replace it with a version of linode.com would look and feel and work correctly, but had one additional feature -- it would collect the login information that people typed in. HTP probably hoped to gain the login for SwiftIRC directly, or collect the logins for Linode admins and obtain SwiftIRC's login from there.
But, before they enacted the domain takeover (a maneuver that would likely be somewhat difficult to employ without being noticed), an HTP member discovered a new vulnerability in ColdFusion, the server software used by Linode. The ability to discover a new exploit on demand implies a high level of skill within the group. Using this exploit, HTP obtained direct access to Linode. They proceeded to gain access to SwiftIRC, as well as other sites hosted on Linode, including a well-known security site, nmap.org
The FBI apparently had a mole in HTP, and they alerted Linode that HTP had access to nmap.org. This posed a bit of a problem for HTP: if it became public knowledge that they had obtained access to Linode, then perhaps they wouldn't have time to go after the impersonators using their newfound access to SwiftIRC. So, HTP tried to strong-arm Linode into staying quiet until May 1st. HTP had obtained the customer information and credit cards of all the Linode customers. HTP threatened to widely publish all this sensitive information if Linode didn't stay quiet. If Linode complied, then HTP would just delete all the info.
Linode, though, was forced by the FBI to announce that they'd been broken into. HTP told Linode to just publicly acknowledge that HTP was the group that broke into Linode, and they'd delete the sensitive info. Linode did so (https://blog.linode.com/2013/04/16/security-incident-update/).
HTP conducted an internal investigation to determine which group member(s) were working with the FBI. HTP broke into the mole's computer and turned on their webcam, and saw an FBI employee looking over the shoulder of the mole. They kicked the mole out of the group, so the FBI doesn't have access to HTP anymore.
(Remember, this is the story according to HTP.)
Here's hoping the FBI "causes trouble" for the lot of them. Breaking into other people's stuff is not cool. If I leave my door open by mistake, yes, that makes me a bit absent minded, or foolish, but it does not give anyone the right to wander into my house.
Just a personal opinion.
" Recognizing their situation, we instead told them that if they acknowledged HTP in their analysis, we'd go ahead and shred their customer data anyway."
Do they honestly, for a single second, think that any LEA, corporation, or, well, anyone would believe that once the information was compromised, that there was no putting the genie back in the bottle? Also - I suspect there are probably disclosure laws that had to be followed by Linode anyways.
Comply, and trust the honor of black hat hackers?
Or refuse, and have customers' data appearing on FTP and torrent sites within the day?
As a Linode customer, I would rather them choose the latter. Not to try and sweep the incident under the rug (to your point about disclosure) but to prevent the data from being scraped by groups who exist solely for extracting credit card details from releases by groups like HTP (note the reference to "carders" in the article) and then being sold.
(According to Linode's own post, the CC data were encrypted, meaning that it should be intractable to actually extract usable CC numbers from the data. But why would Linode not accept those terms, even if they believed that HTP were lying? At least it would give them until 1 May to get their house in order.)
Also makes you wonder, if there are holes like this, how many more holes like this are there? Especially if this is a pattern across the system.
Highlights: 1900+ days uptime on a sparc box somewhere in sourceforge.net, root on ICANN, root on Debian repositories..
- HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes.
- In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though.
- Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators.
- Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April.
- The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons.
- Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it.
- How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access.
So ends clarifications, thus begins conclusions:
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider.
- EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done.
- EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails.
- SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
- Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture?
- HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time.
- Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
:)
The only real solution is either to not depend on any single network, or to make it clear that you will simply kill anyone who troubles you. Or just remain innocuous enough that nobody will care to.
It does have me worried however.
There are plenty of CAs out there who will give out a (non-EV) cert to anyone who can receive emails to webmaster@example.com.
And Linode isn't saying anything new
> We identified which users on HTP were involved with the FBI, and promptly gained access to one of their cams.
Not sure what they mean here. FBI camera? User's laptop camera? Either way this also seems far-fetched.
If everything they said was actually true it's very impressive.
It would be nice if we had internet role models. IRC is full of low-life degenerates who perpetuate the vitriol that reinforces this way of life as an acceptable pastime. If there were well respected hackers who spoke publicly against this kind of behavior it might make some people think twice. (Unfortunately, most well respected hackers used to be these kids before they got real jobs)
HN is full of individuals who try to take the high road, versus the kind of anonymous internet idiocy that exists in nearly every forum and chatroom. I love this about HN. I wish more of the internet took it as an example.
Unfortunately, there's not much slack left to cut. Linode pulled that line taught with the last major breach of their system (and their subsequent opaque response).
It's clear Linode's security practices leave a _lot_ to be desired, and their response to this incident, while better than the last, didn't go anywhere near far enough in terms of transparency (as well as demonstrating some fundamental gaps in their understanding of the current state of infosec).
If they had access to the database, it may have been possible to delete malicious jobs from people's histories. Even if the user had email notifications turned on, an attacker with full access to the database could have turned it off temporarily (just flip a boolean flag).
This whole thing really makes me wonder how many hacks and deals behind the scenes are going unnoticed.
what's stopping the bad guys from just proxying dns queries they don't care about to the original NS?
with this sort of trickery you could get a "domain control validated" https certificate too!
My usual suspicion is that in general, the volume of DNS traffic should give you pause before you start putting custom code in the path of answering a query. Clearly it's possible -- Route 53 is built upon that very notion -- and I suppose in this scenario it's feasible.
Don't forget every Linode has a hostname under linode.com. I think splicing yourself in and running a conditional on every query would overwhelm whatever you point the firehose at and you'd have to plan accordingly. All it would take would be to add a couple hundred milliseconds of latency to the average DNS query (even before the inevitable carpetbombing of p99 latency) and a competent high-traffic administrator is going to start looking around.
I don't understand why HSTS didn't allow some sort of pinning, or the ability to specify a certain kind flag in the certificate is required (like EV).
No, nearly ideal use. Its like a strategic nuclear weapon. You don't use it sneakily, that's the opposite of the whole point. Always intimidate as publicly as possible and in the tech community messing with linode is about as public as it gets.
The other part is showing off, its like declaring "we have access to them all but we don't care about name.com". Maybe they do...
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
Actually, the biggest lesson I'm taking away from this is to never trust any one piece of software, and to always have multiple lines of defense/alerting sitting on independent software stacks.
I wouldn't bet on that.
> There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
Isn't that the same logic Everydns/Dyn Inc. used when they censored Wikileaks?
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
True dat.
I don't need to wager, and can speak with authority based on what I know (which I'd prefer to leave vague). There are two vectors into a Linode's filesystem from the perspective of an internal attacker: having root on the Xen host or gaining a login on the Linode. Knocking over the database and Web server gives you neither unless the person reused their account's password as their root password, in which case it's behind a cryptographic hash and subject to the typical rules there. If you own the database, you do have LISH access which gives you the equivalent of a VGA console; if someone left that console logged in, it's a vector as well.
The only vector HTP would have had in the general case would be bouncing the Linode. It's a fairly sufficient air gap, in a way.
Don't we also have to take them at their word that they even had decrypted CC's? I haven't seen any proof yet that they obtained the decrypted private key, just their statement saying they grabbed in-memory keys.
Domains are jacked and proxied a lot more than people know. The hackers have custom tools (rather than Squid + BIND etc.) that perform these tasks and keep them hidden.
Even better, you could host spoofed DNS for a number of Linode's on a single small 128-256MB virtual machine. The infrastructure required is tiny. Definitely possible, definitely happens all the time.
Keep up with CVE's, don't provide a wide attack area (so lock down interfaces to your machine and don't expose much to the world), and keep blast radii as small as possible (so even if your machine does get owned, you can possibly restrict it so it doesn't automatically mean they gain access to other systems in your network.)
Oh, and model the threats to your network/application. Make sure you're securing against the right threat. As an example, anti-malware is wholly ineffective against social engineering - maybe it's more productive to train employees and make sure that each employee doesn't have total access to all privileged systems.
Damned near every exploitable security problem in an application can be traced back to this one simple rule. XSS is a common one. The ones that don't conform to the rule are rare and magical, like unicorns, and usually involve something exotic like hardware side channels. If you take input it will be abused, so plan accordingly. That occasionally manifests in unexpected ways like timing attacks, in which case an attacker repeatedly sends you lots of carefully-chosen input to deduce something based on how long it takes your code to reply, much like cracking a safe. It's a basic attack on a string compare, which is O(n). This simple code is vulnerable:
def price_multiplier(form):
if form.input == 'super-sekrit-coupon-code':
print >>browser, "100% discount applied!"
return Decimal('0.0')
print >>browser, "Sorry, that's an invalid coupon!"
return Decimal('1.0')
Securing an infrastructure is another story, but the rule is applicable with the occasional clarification to everything there, too. There are domains of trust even within your own organization; malicious employees will try to harm you, as well, and you get little to no warning of those. Do your interns need root on machines containing customer data? Employees are users too.
I just want to know one thing: did or didn't they leak the credit cards?
Does it really matter? If your card was used with Linode, it should have been blocked by now anyway.
Really? You can't think of a terrorist definition that uses, you know terror to coerce people into doing things? While you may not like these guys, labeling it "terrorism" is counterproductive. If you stretch terrorism to include anything that involves the potential for someone to feel fear about anything, you can classify nearly everything as terrorism.
I know hundreds of people who dedicate their lives to the drama and bullshit that is spawned solely by being in an IRC channel. If it went away, these people might just find something productive or positive to do with their lives.
If someone has a beef with society, there are plenty of honest and constructive ways of "sticking it to the man" without resorting to violating people's property.
People say this all the time, but always fail to give examples of what sort of activities they think would be a constructive alternative. Would you mind sharing what you envision as alternative activities?
Best of all, don't have anything worth stealing. Don't keep the credit cards on your servers, parrot them through a vendor. Don't keep user credentials on your system, us OAuth, Fb or Google auth. If you've got nothing valuable to steal, they'll likely not break in.
But then again, if the Fed's want in, they'll just pull your box from the rack. Don't forget that you can literally freeze a DIMM and dump it and all the encryption keys to another mobo. So, as is the CIA's policy, if you don't want anyone to know something, don't let it touch a computer. ;)
The best thing is to keep your eye on the culture of the developers and how seriously they take security - for instance the Ruby on Rails developers ignored exploits/reports until people blew them wide open. Now, if some other hacker had known about that before the disclosure, they could have owned any RoR sites.
From my experience, Django seems to be the best, and has not had any unfixed vulnerabilities for a while (though, due to it's complexity, it's completely possible that 0days exist). However, if I'm running Django sites and some do get owned, I can tell my boss/client/self/whatever that I did everything possible to prevent it happening.
There is no such thing as 100% secure, however, it's fairly reasonable to be hardened to all but the most dedicated crackers.
With an attack like HTP's, there's no fucking way anyone could have been expected to prevent, without running their entire own infrastructure, because they owned registras, Linode's LISH shell (so they get near-physical access to your Linode), and various other crap. If your boss were to fire you for getting owned in this attack, despite it preeetty much being zero of your own fault, they would be in the wrong (unless you have the resources to not depend on anyone).
http://blog.phusion.nl/2013/05/07/phusion-server-security-re...
I've seen that kind of timing attack discussed somewhere, and the solution there was to do some kind of byte-by-byte comparison that would always take the same amount of time. It makes sense.
(R)evolution is not looking at what is wrong, it's looking at what is working. Study the leaders of successful change movements of the past. Most talked about what they wanted (but might not have existed yet), rather than what they didn't want (i.e. "I have a dream" not "I have a nightmare"). Then share that vision with the world. Change begins with a vision of a better and different world. Inspire people to want to move closer to that world. Share your ideas with others. Learn to express yourself through the arts. Form a band. Start blogging. Become a filmmaker. Learn to program and make an app or service that helps people. Build a business that embodies this new world you want to see. Be the change you want to see, said Gandhi.
If you don't yet have a vision for a better and different world, start by asking yourself these questions: "What am I passionate about?" "What's broken with my world?" "What gets me angry or really worked up?" "What are some of my earliest memories of experiencing pure joy as a child?" "What’s the most important thing I know?"
Creating a vision for a better world, sharing that vision with as many people who will listen, and then acting on that vision is the best way I know how to "stick it to the man."
Here's some quotes to help inspire you:
"You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete." - Richard Buckminster Fuller
"People who talk about revolution and class struggle without referring explicitly to everyday life, without understanding what is subversive about love and what is positive in the refusal of constraints, such people have a corpse in their mouth" --Raoul Vaneigem, The Revolution Of Everyday Life
The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. --George Bernard Shaw
And as we let our own light shine, we unconsciously give other people permission to do the same. As we're liberated from our own fear, our presence automatically liberates others. --Marianne Williamson
“Let me say, with the risk of appearing ridiculous, that the true revolutionary is guided by strong feelings of love. Above all, always be capable of feeling any injustice committed against anyone anywhere in the world.” --Che Guevara
“Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has. --Margaret Mead
“Almost everything--all external expectations, all pride, all fear of embarrassment or failure--these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.” - Steve Jobs
"When people focus on the value they provide to others, they take their eyes off their own insecurities." --Dave Navarro
“If you hear a voice within you say, ‘You cannot paint,’ then by all means paint, and that voice will be silenced” --Vincent Van Gogh
"An entrepreneur is someone who, almost artistically, designs a living entity which embodies the values, beliefs, and ambitions of the creator." --Jake Lodwick
"It takes a lot of courage to release the familiar and seemingly secure, to embrace the new. But there is no real security in what is no longer meaningful. There is more security in the adventurous and exciting, for in movement there is life, and in change there is power." -- Alan Cohen
Or, they're lying. I mean, they could be lying about the whole damn story, who knows (not me).
Assuming the account is completely true though, I would say they succeeded in showing off. It's pretty impressive that they took over name.com and discovered a CF 0-day just to get into Linode, all just to fuck with one IRC network no one cares about.
If I can't trust you at your word, you no longer have the privilege of holding my data or my CC info. Two breeches with a lack of communication really does spoil overnight the trust that has been built up over years of wonderful and reliable service.
This security incident is very upsetting, but for me the linode track record in communication, responsiveness and support is still pretty amazing compared to the competition. At least at this price range.
AWS might be safer, but I basically don't get any support (for a reasonable price like I pay linode), Rackspace didn't seem nearly as responsive or transparent on much more trivial matters. Who else is out there which is worth switching to?
(I'm not being sarcastic, I'm genuinely curious)
I fail to understand how you can think Linode has a great track record. It is an unequivocal disgrace. TWICE they have mislead their own customers over a major security incident. And there have been lots of times during outages (in particular my time at Fremont) where they were MIA.
I'm merely a hobbyist/researcher, so I don't have a production environment to worry about.
ISTR being able to back up Linode images, migrate them around, and perform some other admin tasks from the web interface. So I still wonder if the "air gap" API is a bit more powerful than simply the ability to request a reboot. But, again, I don't know.
Without knowing what those things are, I'll just take your word for it that none of them could ever possibly be leveraged to compromise a host or guest without unmaskable and permanent messages appearing in the logs.
I won't give my email or its password to you, but if you can find it, hack it and decrypt my emails, then it would be only my fault, and you will have my respect.
"Fault" is not in question here either. Let's say I leave my front door unlocked. If you enter my home without my permission, you have trespassed and can be charged with a crime. The only thing I would be "at fault" for is making a lackluster attempt at securing my home. I don't forfeit protection from trespass under the law for that act though.
You see, locks are not what govern access; laws are. HTP is clearly in the wrong here. They forced entry in to Linode's systems, then attempted to extort Linode in an effort to achieve their goals. Swap out Linode's servers for Linode's offices, and there's no question that HTP are operating outside the boundaries of ethical behavior.
Ah yes, the "might makes right" philosophy that we all know and admire from primary school.
The US/Israel are involved in a proxy war with Iran that involves cyberwarfare, clandestine operations and conventional military strikes (in the case of Israel striking Iranian-sourced Syrian weaponry).
It's an extremely poor example to use open cyberwarfare between nations engaged in everything but overt warfare to attempt to legitimize black hat hacking.
And then collected private information of everybody who works for or is a customer of the institution, and then threatened the owners of the institution if they spoke out about what had happened.
I think quite a number of people tread this fine line between approving of sort of Robin Hood types and criminals. And that can change given the target. So, "fantastic" if they embarrass the FBI, not so comfortable if its our bank.
Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it. And you're still going to introduce latency by inserting another hop. That's my point, though I do agree with you.
Welcome to the world of recursive name servers, there is a lot of software out there that does exactly what you just mentioned, I fail to see what would be hard about making this change.
Hm, why? Any modern CPU is blazingly fast. Writing it in Ruby probably wouldn't be smart, but Python + PyPI or Lua + LuaJIT would easily get within a factor of 10x of C.
(I mean, "More about governments than organized crime? There's a difference?")
They never ever disclose when they get hacked. And they do get hacked.
You can say the same thing about SMS, Facebook, Twitter, IM in general, email in general, MUDs and MMOs, telephones, telegraphs...
Should all those go away, too?
What? I could name 20 channels of the top of my head this is not true for.
IRC is nothing like you've described, at least to me.
So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely), or the hackers must have had control of the machine during the time in which they did billing (which I don't think is true, because billing presumably happens either at the start or the end of the month, and didn't the hack take place a bit earlier than that?).
So yeah, I believe them when they said they got the private key. But nobody's said anything to convince me that they got the _decrypted_ private key. And if the passphrase really is as long and complex as Linode claims, then it should be reasonably secure (caveat: I am not a security researcher, or otherwise qualified to judge the security of anything).
I need to be able to serve my customers, and they are all over the world. So I need a provider with data centers in multiple global locations, that offers an API, that has decent support that responds quickly, and at a similar price range... Interested to find alternatives that are also more secure and better at communication.
User hderms didn't describe them, I did. And I can show you hundreds of thousands of channels like I describe, on most popular IRC networks (Freenode is a tiny network in comparison).
Also, I challenge you to show me proof of intelligent discourse in any Freenode channel. It's simply not easy, especially in a channel with 5 or more active participants in conversations. Try taking your time to make intelligent points and either people get bored with you or your points get lost in the scrollback.
But seriously, yeah, of course there's a difference, but government is also often in league with organized crime, from local levels of corrupt cops and drug leaders, to international levels of the CIA and arms traders and 'our' 'freedom fighers' -- I bet this is also true in Sicily and Naples, that many parts of the government act in league with organized crime.
But yeah, to go back to the actual topic/thread, mainly, I think the guy is right that the government is the other main actor which is going to keep it's security exploits to itself, and just use them to monitor you silently. (And that's probably more likely in the US than any other country on the planet, at the moment).
Italy is not known for having particularly good politicians or government, but I'd take the government here any day over the Mafia or Camorra. Despite any romanticized movies you may have seen, those guys are pretty cold-blooded SOB's. Recently, a man was shot dead waiting to pick his son up from pre-school in Naples.
They bill you the moment you add a Linode, automatically, if your credit is not sufficient to cover the new Linode. Careful walking that assumption too far; I think it's safe to say the key was kept in memory.
Good move on their part.
In an ideal world, yes, they should be more secure. However, as in this case, they got taken advantage of via a zero-day attack, with others planned well outside the scope of what Linode could have planned for. Which is insane. Can you even name something, anything that they could have done to protect themselves? Additionally, given the unique form of attack, figuring out what was going wrong was probably not possible. Thus, they knew as little as you did.
And then, everybody switches to some other provider. But do they switch to "super secure, we examine every byte of the software that we run to make sure we're bullet proof" hosting provider? NO, everyone just switches to another commodity VPS provider that is vulnerable to all the same super high level attacks that Linode is vulnerable (maybe even more attacks, given that Linode actually has a tremendous amount of experience).
In reality, you're only getting more security by switching to a less prominent hosting provider, A.K.A. security through obscurity. Which is the worst kind of security because it's not secure at all.
It's like getting mad at the mayor of your city when a meteor falls on your house: unproductive and misguided.
I'm just as annoyed at Amazon for this, to be honest, and in the large, annoyed at our industry for being so unnecessarily secretive. We need to stop thinking of our infrastructure as our competitive advantage; to pick on Google as an example, while Google are obviously masters of running systems at scale, their infrastructure efficiency is not the reason people choose Gmail. Obviously their platform gives them some competitive advantage but, for example, their policy of withholding even the innocuous names of internal systems is bizarre. I think the rest of the industry follows that lead.
It's weird that we embrace openness in the FLOSS communities but when it's time to build a revenue-making company, the details of the inner workings are immediately a hush-hush secret. If you're doing something simple enough that describing it means someone can replicate it, it's an idea that can be replicated trivially anyway. I bet everybody in hosting knows how Linode works, and I doubt there's any kind of espionage taking place.
In this case, it's fine to be secretive if you'd like, but at least tell me how you plan to prevent the problem from recurring. Linode always says "we're working diligently to prevent this from happening again" but provides no details whatever. The announcement from the founder of Linode[1] underlines this; the entire tone of the post is "here's how we band-aided the immediate problem," with no details on where they go from here as a business or culture.
When a security incident happens, I believe most security professionals would advise to keep details to the minimum necessary. I can imagine how misleading info can cause panic and dire consequences (to both linode and its customers). In Linode's case this could have been mandated by the FBI even, giving Linode no choice.
For me, linode is still one of the more transparent providers out there. I doubt AWS or any other provider would be more forthcoming if something similar happens.
Of course, there's a lot of security improvements to be made. I hope Linode would shake-up and improve and signs are they're doing that.
I'm still curious to hear some brand names that are better in that respect (hence my question about). From what I read there really is no better alternative currently at this price range.
This isn't a "could it be done?" exercise, it's more of a "could it be done without detection?" exercise. For this specific case, it's a pretty big risk.
As for detection, that was the reason I brought up CPU power. Modern CPUs are so fast that that it seems like this redirector would hardly generate a blip in any chart (such as top).
I don't care about proving anybody wrong. I care about filling my knowledge gaps. I.e. it's interesting to try to figure out how something like this would be detected in practice.
Who is going to notice increased latency in DNS queries? Most likely web developers. Nobody else I can think of would do (non-cached) bulk DNS queries to random domains and actually be looking for millisecond changes in lookup time. And those developers would have no insight to the DNS infrastructure serving requests, so they'd have no idea to contact the DNS admins to investigate. Even the DNS admins could be fooled before they contact network admins to do further research.
The bottom line is not "has DNS latency changed?", it's "has DNS latency become unacceptably high enough to force me investigate?" Unless it's becoming a problem, I think anyone would ignore increased latency because they have ten other work tasks to deal with.
Think about the dumbest way you would do that. Then implement it. That's how simple our system is.
The main goal of my comments is to object to the opinion that hacking is somewhat comparable to physical break and enter actions. This is an age where one can find himself in prison for tens of years for hacking and getting access to information (the prospects of Aaron) or even for IP violations, as it were a murder or rape.
Being in an underground hackers crew is much fun and possibilities to learn things for young men who are smart and different than their friends. Those guys and gals are the future top-class engineers at Google and other IT giants and I want them to continue hacking and growing personally and professionally, not rotting in the prison.
For example, poorly written laws make even simple port scanning a risky activity. I agree that this is ridiculous, but you needn't defend HTP here in order to take that position. If anything, HTP are to blame for the overreaction from policy makers. They are having a very difficult time distinguishing between mischief and mayhem.
The law should take context in to account. If you were caught exploring an old, abandoned warehouse, you might end up with a misdemeanor trespass charge (hopefully), but if you're caught probing a corporate server, the current climate seems to dictate that you'll land a felony in short order. You've got groups like HTP to thank for that because of their extortion activities and willingness to leverage the well being of innocent people in their trivial games.
Or the next members of the many criminal syndicates on the internet who steal/harass/blackmail with little regard of the consequences.
I had a quick look on the Wikipedia page but couldn't see anything there.
That's really besides the point though. Anyone who seeks to boil "right and wrong" down to a simple principle that applies in all cases is holding simplicity above practicality. "Simple principles" are no more sustainable than utopian ideologies. The reality is that right and wrong are subtle and contextual. Bringing Stuxnet in to the Linode discussion is just an effort to sidetrack the topic.
Any way you slice it, HTP has dragged "innocents" in to the fight. It's not right when governments do it on the world stage. It's not right when thugs and gangsters do it by terrorizing neighborhoods with gang violence. It's not right when crackers do it during their internet turf battles.
1: http://en.wikipedia.org/wiki/Mahmoud_Ahmadinejad_and_Israel
I certainly agree with you about the actions of HTP being unethical.
You'd be surprised once you start working with larger, higher-traffic infrastructures. If our average external DNS query rises 200ms, my phone goes off. There's more slack on p99, but it's also monitored.
All of the timings for the various parts of a request to the system that I administer are instrumented from a small libcurl app running in multiple ASNs remotely, because Pingdom and other services do not provide the resolution that we need. They are then rendered on a stacked graph that always lives on my third monitor, and any significant deviation averaged out over five minutes catches my eye.
I know it sounds like overkill, but it's crucial at scale.