Security Hole in Sendgrid(chunkhost.com) |
Security Hole in Sendgrid(chunkhost.com) |
The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.
Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor.
Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).
EDIT: May I ask what can be gained from faulting SendGrid in this case?
They literally handed over a customer's credentials over the phone. What's more, there didn't appear to be any ID verification.
I don't see how that could be anything but a huge, glaring, faultable security hole. I'm a Sendgrid user, and this is pretty scary.
So if social engineering is possible, blame the software architects.
Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.
This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify.
But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."
"Massive Security Hole in ChunkHost. Non-2FA accounts can be owned."
Because it turns out anyone with a Sendgrid Support account also effectively had potential access to any account at ChunkHost not using two-factor authentication. Which is also true of thousands of other companies that are relaying their password reset emails through third party SMTP services.
SendGrid seems lame, for allowing this and for their response promising to yell more loudly at their support people, but they're an SMTP relay service not an authentication service.
I thought that was just the strangest thing, and it didn't sit well with me. Accessing a customer's account when they request service is one thing, but making changes on behalf of a stranger you know has no authority over that account? That was when I moved the last of my apps over to Mandrill.
This is the part that scares me. Do they not have auditing in the system where the representatives are able to change the email address on file?
1) You sign up, enable two-factor auth, then lock yourself out (lost password and your second-factor). How do you prove to the service provider that you are you?
2) You sign up, enable two-factor auth, then Mallory claims that they locked themselves out. How does the service provider prove that Mallory is not you?
Specifically--can AWS support staff grant access to AWS accounts, and if so:
what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited?
As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner.
The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern.
[1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.
Trust me I know that CC's are getting sniffed in transit too often so I'm not saying they're 'safe'. I'm just wondering if there is something unique to Bitcoin that suddenly makes you a target as though you were known to be storing CC's data at rest onsite.
I'm going to bring this up with our team and see if there's another vendor that can more reliably protect our customers.
Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs.
And you're going to move away from them now?
For example, why should 1st level support have the ability to make major changes like this? It sounds like only 2nd level support, a smaller group of more highly trained support staff, should have the ability to do this. Does SendGrid have enough money/resources to split their team into 1st and 2nd level support? Would a larger company have those type of resources that would better protect my customers?
These are questions I will talk over with my team.
I mean, keep in mind that this is the same company that publicly crucified a female employee in order to stop a DDoS attack. Clearly there are some priorities out of whack there, and given the insecure nature of e-mail in the first place, I would never want to deal with a company that is so clearly unprofessional.
1) If your support chat doesn't enforce authorization, always ask the requestor to send you an email. 2) Make sure the domain is correct (that's where Sendgrid screwed up). 3) Never agree on replying to a different email address than the one of the sender.
Should read "social engineering resulted in security breach" and guess what, this happens all the time whether sendGrid or not.
Or is it that easy to register a domain with a fictitious persona?
Virtually every registrar allows instant update of whois records via a web interface. I can have Bill Gates be the technical contact on my personal domain in about a minute.
I suggest we dub the act or state of being one's own worst enemy, being "sendgriddled."
It's not like it is hard. At least not harder than integrating to a third party email sender.
- You need to make sure that your email servers IPs are not on black lists.
- That you use DKIM properly
- your multipart mime encoding is correct
- bouncing e-mails are handled...
- take care of scaling & operating the servers
- and so on....
You can spent ( and waste ) a lot of time on this... especially if you are new to email sending..
I've recently tried to use their service to send emails, club member newsletters. I've never thought much about bulk e-mails before and I thought it was a "solved problem" by now. Sendgrid are well known so they were my first choice.
During initial testing I found both bugs in the API and missing functionality. I've worked over 20 years with IT and yet sendgrid support is easily on the top-3 list of my worst support experiences, a total waste of time.
We ended up using mailgun instead and so far it looks much better.
Overall they've been a good service provider and their tech support has always been helpful and pleasant to deal with. (I know! Crazy, right?).
I use Mandrill for my private projects, and it is so much better it's shocking SendGrid has a single customer.
You're right, that model is deeply broken if anyone can intercept those emails (as happened in this case), but it seems unfair to single out ChunkHost for criticism.
The thing is we trust these companies to have processes in place so that their representatives won't have the ability to potentially do something destructive and if they can because they are the highest ranked rep and they need that kind of access - then at least auditing and training should be in place to avoid that kind of behavior.
I'm not saying this is fair.
1) You decide how valuable the account is, the probability that you will lose access to the account, and the probability that the account will be attacked. 2) You selected the required number of recovery actions, from one recovery action to completely unrecoverable. Possible recovery actions include (copied from NFSN):
* You provide a scanned copy of a government-issued photo ID. * You provide a scanned copy of a statement showing both the most recent deposit and a name and address matching one of your accounts. * You complete SMS verification. (SMS must be previously configured.) * You complete 2-factor verification. (2-factor auth must be previously configured.) * You correctly answer your security question. (Security question and answer must be previously configured, below.) * You use an ssh key to create a file with a specific name on one of your sites hosted here. (Must be previously configured, won’t work if account is empty.) * We try and fail to contact you via your currently configured email address. (This one may take a long time.)
As far as I'm concerned, this is the way it should be done. The public details are on their blog: https://blog.nearlyfreespeech.net/2014/02/28/price-cuts-more...
This alone is a huge, huge chore. Especially if you run a hosted service that allows some user-specified content in outbound email bodies. It's a lot cheaper for us to pay Mandrill to handle all of that for us, provide us excellent metrics and diagnostics, and let us know if one of our users is sending junk mail before it gets out of hand.
Yes, sending an email is easy enough. It's all the other stuff we have services like this for.
http://aws.amazon.com/ses/pricing/
"You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk."
Actions SendGrid could take:
* Make it impossible for their front-line support staff to change the email address on file. If you want that -- which should be extremely rare! -- you talk to a high-level manager who is competent at authenticating you.
* Send the email that says "hey, we're going to change your email address now" with a lead time to allow for the possibility that, even after your authentication, you've been conned.
* Make a phone call to the phone number on record, too.
You ask what's gained by faulting SendGrid, because you take it as a given that they will make these changes. But that's not how blame works. The blame serves a function of ensuring those changes by holding them accountable for their current problems.
Or perhaps they should allow customers to require that they contact a secondary authority to confirm the change should be made?
If you omit it, the assigned tech will frequently ask for it. Always sends a little shiver down my spine.
After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill a few months ago and have been very impressed. It's still a tiny, microscopic percentage of our budget, but we get so much more (open/click reporting, sub-accounts, rendered email body history, much better blacklist/whitelist management).
My employer has been making more and more use of SES, and we've just started looking at automated processing of feedback notifications [1]. Did you find them lacking?
[1] http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notific...
They get accurate data for clicks, but geolocation/browser stats for opens aren't 100%. They still track the rates, but it looks like they're coming from one of Google's proxies, so this is the best they can do for now.
Blacklists suck. Not as much as spam sucks, though.
Maybe they FedEx the password to your physical address on file. Maybe they contact all phone numbers and emails they have for you and say "someone has requested an emergency override, if you object call us back in the next 4 hours." Maybe they do a Skype session and compare your photo to the one they have on file.
All this costs money, of course. That's the price of doing business.
Do you tell the user on signup to print an in-case-of-emergency-break-glass password which is only ever to be used to get into a locked account and other special circumstances?
It may seem over the top but seeing as it's unique across service providers, I think it's a hell of a lot better than the overly abused "what is your mother's maiden name" type questions. I consider these questions to be in the same boat as sharing passwords between websites (since they are)!
(And if someone has managed to break into both your personal email account and your business's online-banking account, getting your web-host to recognize you will be the least of your problems.)
Consider a determined attacker. A posted signed letter has zero cost and is easily forged and a phone call is free via Skype. There's plenty of low-tech ways to circumvent security.