Segment Security Incident(segment.com) |
Segment Security Incident(segment.com) |
That implies that Segment employees, or a subset, have unfettered access to view their customers' accounts? That it didn't require positive customer assent to gain access?
I like Box's model for this: https://community.box.com/t5/Working-with-Box-Support/Box-Pr...
Here is the email I got:
" We are writing to notify you of a security incident that happened between August 26 and August 31, 2019. We became aware of it on August 31, resolved it immediately, and have been assessing the scope of impact since then. Based on that ongoing assessment, we have concluded that the incident involved your workspace. We are very sorry for any issues this may cause.
What happened? Between August 26 and August 31, 2019, an unauthorized party compromised a Segment employee’s account and used it to gain unauthorized access to Segment product usage data. Upon detection, we took immediate action, disabling and deleting the account and removing unauthorized access. We also reported the incident to law enforcement.
What information was involved? The unauthorized party accessed information about how Segment users interact with the Segment product, as well as first name, last name, email address, and IP address for your Segment users, and the Segment write keys for your workspace. No Segment customer passwords were compromised.
No personally identifiable information relating to your own customers was accessed. As a result, no action is required from you at this time.
More information For more information, please visit oursecurity bulletin page. This page contains a detailed timeline about what happened and information about what actions Segment has taken in response. It will be updated with any new developments.
If you have any further questions, please reach out to support@segment.com. We again apologize for any inconveniences this incident may cause.
Sincerely, Coleen Coolidge Chief Information Security Officer"
P.S. I got that email, too
It's an interesting exercise to imagine an elaborate multi-custody arrangement where all data is encrypted with keys that live in hardware security modules and at least three of five senior people are required to join their secrets any time the KEK needs to be handled directly... but I can tell you that that kind of scheme where 100% of customer data is completely opaque would be a major inconvenience in a fast-moving SaaS startup. Not only is implementing an engineer-proof privacy scheme a massive undertaking but it makes debugging production issues much more difficult. If indeed they were going to go all-out on such a scheme I would kind of expect them to brag about it but I don't see claims like that in https://community.box.com/t5/How-to-Guides-for-Account/Data-....
I'm curious if anyone's worked at big-name cloud services providers and can speak to the practical effectiveness of privacy measures in place. My guess is that even at Google, for example, a rogue engineer working on a particular product (or maybe the infrastructure it runs on) could theoretically bypass organizational controls and look at user data.
To be clear, I don't really view this state of affairs as a big problem. I've commented before on why (https://news.ycombinator.com/item?id=20513055). Professionalism and integrity (in combination with organizational controls and the principle of least privilege) work fine IME, plus the fact that nobody except you really cares about your personal data. But I do think that your expectations are unrealistic if you think that backend engineers generally don't have the kind of access I'm talking about. Unless a service encrypts on the client with your encryption keys, you should expect that the company's "employees, or a subset, has unfettered access."
Ideally the private key for the VPN is a hardware token, but I will admit in most cases it’s simply a file on the drive.
The Admin Panel has zero third party JavaScript nor client-side analytics.
I’m not sure this buys quite as much security as it might appear, because there’s probably any number of ways to piggyback on a valid session.
For example, there was a bug bounty Tesla paid out when a researcher was able to establish XSS by mangling their car name, which was read via the API and showed up in an internal dashboard as unescaped HTML.
So I think the biggest attack surface for Admin Panels remains a hostile client seeding XSS into unescaped fields, and CSP helps a great deal with this, but at the very least I don’t see any reason why these panels should be on a public URL.
You can say, well, they should have MFA on the panel, but I can’t shake the feeling that these simple measures avoid a huge number of attackers looking for low-hanging fruit.
It’s like putting SSH on a non-standard port. It’s theoretically meaningless but practically it’s a huge improvement, if only because it helps attack signals stand out against the script kiddie noise.
At which level the interface between admin panel and apps is? Is it a dedicated app for the admin panel and just a front end to an AD or database? Like there is no real admin panel on the app itself.
They share much of the same underlying data access and business logic code, but the screens that let you see all the clients, change SKUs, monitor billing, impersonate a user, generate reports, etc. just would not exist on the public-facing site.
Underneath both sites/apps are talking to the same database(s).
Has the risk of of other Segment accounts having been compromised through the same channel (but have yet to be detected) been investigated?
I hope not, since there's a 5 day gap between CircleCI being notified and the rest of their customers.
Even if it was though, Circle discovered this issue on their own through an automated notification of an action taken; they do not seem to have been notified directly by Segment. It doesn't seem fair to assume that _if_ Circle is referring to Segment, that either company did anything wrong in their response here.
> For a small subset of customers (13), the unauthorized party was able to gain read-only access to their workspaces and click around in their accounts for up to a few minutes. These customers have been notified.
I believe this is critical. Access controls really should be set as needed, per user, and fine grain. While it can be time consuming and may not even make sense at the beginning of a project, doing everything this way from the start should help to keep an organization better secured in the long run.
This should always be step #1, don't wait for an incident.
We've surveyed startup dir/security's (and the like) and this is almost universally in everyone's top 3, and the leading contender for #1.
Information about how Segment customers interact with different aspects of our product,
*including customer write keys for Segment*, integration names, workspace names,
and how customers interact with our user interface.Since this was a privileged account, how can they be sure the attacker didn't use said account to setup more ways to get back in? That's the first thing Kevin Mitnick always did when he pwned a box: setup alternative routes to get in, in case his original door got closed.
It always seems to be the marketing analytics data that's wide open for 'hackers'.
You could spend all day building a secure DB and application architecture then have the marketing team upload analytics for everything onto some random insecure service.
Maybe the marketing/data teams need to get some security lessons as part of their training the way programmers learn?
Why all the down votes... this is a legitimate security issue.
1) Your idea has been shown not to work (it's called perimeter security) 2) With the huge shift to cloud services you need a new solution anyways
I would like at least one company to post an "incident" reveal in a more honest way:
" Due to our carelessness and relatively insecure practices, we had improperly disclosed user accounts to a moderately savvy hacker. We realize this is our fault.
If you'd like to help and given that we have your attention now, it would be valuable if you can help pentest our servers: the attacker used a simple SQL attack based on an unpatched server via CVE-3245. Are we missing anything else? Please let us know.
Thank you."
Good security is not easy, and not always due to "carelessness".
It's an expensive, onerous, never ending, and ever evolving process to get right. Most, if not all, companies do the bare minimum security they believe is necessary; anything beyond that is a waste of money and computing resources (if you believe otherwise, I have some retina scanners to sell you...)
This why we continue to have incidents and vulnerabilities which could have been prevented, or better mitigated. Most often these companies do not even know how to make a correct assessment of their risk. They move forward with this idea that it's a waste of money and resources, yet waste everyone's time with clean-up, or just go out of business as a result. Even with minimal security training and limited curiosity, this incident could have been avoided.
MFA with a U2F token would go a long long way. Even softU2F as github built might prevent this.
They weren't using 2FA, and only enabled after this incident. This is 100% Segment negligence.
At least my adblocker (browser) did not like what it saw and blocked the article.
What happened?
Between August 26 and August 31, 2019 an unauthorized party compromised a Segment employee’s Segment web application account without their knowledge, logging in with their email and password. This account had privileged access.
Using the employee’s account, the unauthorized party acquired two months of recent data relating to how Segment customers use the Segment product. This includes information about how Segment users interact with our application and associated account information (email address, first and last name, IP address for each session, and Segment write keys). No Segment customer passwords were compromised.
This data is used by our product, marketing, and customer success teams to provide ongoing support for our customers.
When did Segment discover the issue? What did Segment do when it discovered the issue?
We learned about the incident on August 31. Upon detection, we took immediate action, disabling and deleting the account that was compromised. We then began a full investigation to understand and assess the impact of the incident.
What information was involved?
Over the course of our investigation, we learned that the unauthorized party acquired two months of recent data relating to how our customers use their Segment workspaces. This includes:
Information about how Segment customers interact with different aspects of our product, including customer write keys for Segment (which are considered public), integration names, workspace names, and how customers interact with our user interface. Information related to Segment customer accounts including first name, last name, email address and IP address while using the Segment web application. No Segment customer passwords were compromised. For a small subset of customers (13), the unauthorized party was able to gain read-only access to their workspaces and click around in their accounts for up to a few minutes. These customers have been notified. What is Segment doing to ensure this doesn’t happen again?
We have taken immediate action and are continuing to investigate and assess the impact of this incident. Upon discovery, we:
Enforced mandatory Multi-Factor Authentication (MFA) for all employees when accessing Segment-owned workspaces and performing administrative actions in the Segment app. Reset all employee sessions and passwords as a precautionary measure. Notified relevant law enforcement authorities. Removed privileged access controls internally, adding accounts on an as-needed basis. Began an audit of internal access controls to mitigate the risk of an event like this happening in the future. What should I do about this incident?
Unless you have been specifically instructed differently by Segment, there is no direct action required. However, this is a good reminder to make sure that your Segment password is a unique, strong password. We’d also recommend you activate Multi-Factor Authentication (MFA). This blog post covers strong passwords and activating MFA.
We apologize for any inconveniences this incident may cause. If you have any further questions or concerns, please do not hesitate to reach out to us at support@segment.com.
"Based on that ongoing assessment, we have concluded that the incident involved your workspace."
If it's not hardware-based (i.e. Yubikey), you can still spearphish people into putting their username, password, and 2FA token into a honeypot page which would give the attacker a window of unauthorized access.
No matter how good security gets, attackers will always adapt. Everyone on earth is now using YubiKeys? Now you need a process in place for when people get their keys lost or stolen. Or when your computer doesn't have a USB port. And whatever "I forget my password"-esque process that is will probably be much easier to attack/manipulate/social engineer than the keys themselves would be.
Other changes are mildly annoying to developers, ops, and support (e.g. re-requesting production access.) Since developers hold sway in most organizations, convenience often trumps security. None of these measures put anyone out of business.
If I had to attack something I'd go for the limited resources to help smaller organizations scale security appropriately. There are tons of resources for large dev teams, infosec specialists, etc., but there is very little that targets small organizations effectively.