Thanksgiving 2023 security incident(blog.cloudflare.com) |
Thanksgiving 2023 security incident(blog.cloudflare.com) |
Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases.
Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting?
TA hijacked a session token of a CF employee from a support ticket.
Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform.
How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA?
Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code.
Did the stolen employee credentials only have access to Atlassian?
Blog: TA gained access to a set of credentials
Does this mean multiple credentials got uploaded to the Okta support system?
Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November.
Does this mean the compromised token was long-lived?
Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.
I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise?
Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system.
Is this service token a bearer token? And without expiry. Is this like an API key?
TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway.
2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance,
So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry
TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment.
Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks.
3. A Bitbucket service account, which was used to access our source code management system
4. AWS environment that had no access to the global network and no customer or sensitive data.
Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA.
The only production system the TA could access using the stolen credentials was our Atlassian environment.
Mitigations:
Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.
What does hardening security protocol mean here? Is it techniques for D&R or something else
Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials)
I believe this means forced resets on employees (Okta users), right?
> Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories
> They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).
Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.
> Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin.
> This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL.
Ouch. Full access to a server OS is always scary.
It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc…
Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.
2M tickets - in my 4.5y at present company we've probably averaged about 10 engineers and totalled 4.5k tickets. Cloudflare has been around longer, has many more engineers, might use it for HR, IT, etc. too, might have processes like every ticket on close opens a new one for the reporter to test, etc. It sounds the right sort of order of magnitude to me.
Source: former Cloudflare employee
They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 194,100 pages)
I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand?
2M Jira tickets is probably easily achieved if you create tickets using any automated process.
Yes, they aren’t perfect. They do some things that I disagree with.
But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this.
Thank you for the blog post!
- Insist that you have better integrity than your competitors
- share a few operational investigations after your latest security event
what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.
They mention a third-party audit was conducted, but thats not because they care about you. Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information, it needs to pass a yearly on-premise audit to ensure PCI compliance. if they didnt, major credit houses would stop processing their payments.
It's a very good writeup, as these things go. Cloudflare is huge. An ancillary system of theirs got popped as sequelae to the Okta breach. They reimaged every machine in their fleet and burned all their secrets. People are going to find ways to snipe at them, because that's fun to do, but none of those people are likely to handle an incident like this better.
I am not a Cloudflare customer (technically, I am a competitor). But my estimation of them went up (I won't say by how much) after reading this writeup.
They clearly defined the scope of impact, and demonstrated that none of this impacts systems in scope for PCI. There was no breach to change management inside of BitBucket, and none of the edge servers processing cardholder data were impacted. They will have plenty of artifacts to demonstrate that by bringing in an external firm. So I am really not clear why you're bringing up PCI at all here. They made it clear no cardholder data was impacted so your perspective on the required "on-site" audits is moot.
Cloudflare operates two entirely different scopes for PCI; The first being as a Merchant where you the customer pays for the services. This is a very small scope of systems. The second is as a Service Provider that processes cards over the network. The network works such that it is not feasible to exfiltrate card data from the network. There are many reasons as to why this is, but they demonstrate year over year that this is not something that is reasonably possible. You can review their PCI AoC and get the details (albeit limited) to understand this better. Or you could get their SOC 2 Type 2 Report which will cover many aspects of the edge networks control environment with much better testing details. After reading that you can then come back to the blog and see that clearly no PCI scoped systems were impacted in a way that would require any on-prem audit to occur.
And they are not a card network. They are a PCI Service Provider because cards transit over their network. They are not at risk of being unable to process payments or transactions for their Merchant scope even if there are issues with their Service Provider scope. Because, again, these are two separate PCI audits that are done, testing two different sets of systems and controls.
And, as an aside, Cloudflare effectively always has on-prem PCI audits occur. Because the PCI QSA's need to physically visit Cloudflare datacenters to demonstrate not only the software side of compliance, but the datacenters deployed globally.
They did, and they admitted that it was their fault. I have to give them credit for that much.
> They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023...The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
Like I get cynicism, but they very clearly explained the lead-up to the accounts being compromised and the mistakes that caused that. They took full accountability of it. Which is frankly more than most companies dealing with security incidents. This entire write-up is more than most companies obligations or responses.
No competitors were mentioned in Cloudflares article, they explained what kind of information was breached, nothing to do with payment/card info... so I doubt you even read past the first few paragraphs/conclusion.
No payment card information was compromised.
I don't believe they didn't lose anything. That's not how this works, and most Jira/Confluence I've seen is loaded with secrets.
For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company.
Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.
I'm curious if they're rethinking being on Okta.
Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?
> The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.
https://github.com/BishopFox/sliver
"Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin."
https://blog.cloudflare.com/thanksgiving-2023-security-incid...
obviously a customer data breach would be worse but this is really no bueno
The customer data next year is not the same as the customer data this year.
In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.
This odd to me - unused credentials should probably be deleted, not rotated.
1: "what are these accounts?"
2: "oh they're unused, they don't even appear in the logs"
1: "we should rotate them"
2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons"
?
But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.
Am I missing something here?
There’s no machine cert used? AuthN tokens aren’t cryptographically bound?
This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”
If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.
It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain.
These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now!
Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack.
Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius.
[1] https://docs.github.com/en/actions/deployment/security-harde...
The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data.
So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach".
You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.
Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...
EDIT: re-opened the link a few minutes later and now I see the post
The thoroughness is pretty amazing
Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise.
Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.
Okta hitting everywhere
This seems incredibly wasteful.
Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.
> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers.
It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad.
Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.
Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.
Why didn't they start this effort BEFORE there was an incident?
> we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials
Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this?
> To ensure these systems are 100% secure
Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag.
> Nothing was found, but we replaced the hardware anyway.
Well that is just plain stupid and wasteful.
> We also looked for software packages that hadn’t been updated
Why weren't you looking for that prior to the incident?
> we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials.
And yet they continue using Okta. The jokes just write themselves.
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.
Wait, wait, wait. You KNEW the accounts with remote access to your systems were UNUSED and yet they continue to be active? Hahahahaha.
> The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.
Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.
Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.
The rule does not set any specific timeline between the incident and the materiality determination, but the materiality determination should be made without 'unreasonable delay'.
They had government engagements with Google's consent, and all those various engagements could be disclosed to each other?
If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?
Australians get the 'opportunity' to be part of that sort of that sort of espionage as a base level condition of citizenship [0].
As an upside, I guess it helps with encouraging good practices around zero trust processes and systems dev.
[0]: https://en.wikipedia.org/wiki/Mass_surveillance_in_Australia...
Precisely. Particularly in the case of US businesses. Why bother picking a lock when you have both the key and permission?
Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat.
The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.
Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.
January 2022: https://blog.cloudflare.com/cloudflare-investigation-of-the-...
October 2023: https://blog.cloudflare.com/how-cloudflare-mitigated-yet-ano...
If anything Okta is a bigger company (by revenue, by employee count) and they were founded a year earlier.
It's fair to "punch down" imo as that's how the credentials were originally compromised. I'd agree with you if CF were trying to minimize their own mistake but that doesn't seem to be what is happening here
I am grandfathered in to an old MacBook that has absolutely no management software on it, from the “Early Days” when there was no IT and we just got brand new untouched laptops.
They offered me an upgrade to an M1/M2 pro, but I refused, saying that I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.
Since that would hugely disrupt my work, I can’t upgrade. Maybe I can use incidents like this to justify my beliefs to the IT department…
Okta doesn't make device management software, thats made by companies like Jamf. Okta can integrate with them but Okta isn't what manages your laptop at all.
> I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.
Do not do this, its not a personal device.
Well... don't do that? Why would you ever have personal anything on a work computer?
That is, I'd expect there was a flag in a database somewhere saying that those service accounts were "abandoned" or "cleaned up" or some other non-active status, but that this assertion was incorrect. Then they probably rotated all the passwords for active accounts, but skipped the inactive ones.
Speaking purely about PKI and certificate revocation, because that's the only similar context that I really know about, there is generally a difference between allowing certificates to expire, vs allowing them to be marked as "no longer used", vs fully revoking them: a certificate authority needs to do absolutely nothing in the first case, can choose to either do nothing or revoke in the second case, and must actively maintain and broadcast that revocation list for the third case. When someone says "hey I accidentally clobbered that private key can I please have a new cert for this new key," you generally don't add the old cert to the revocation list because why would you.
Great call out too
> Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.
Because if you take it exactly as it's written it's just too weird, I'm not a security expert with something to teach Cloudflare about err maybe don't leave secrets lying around that aren't actually needed for anything, that's not news to many people, and they surely have many actual security people for whom that would not even be a fizzbuzz interview question reviewing any kind of secret storage or revocation policy/procedure. And also the mentioned third-party audit.
Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.
Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.
Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.
It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.
Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.
Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.
(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)
I can just imagine the attackers licking their lips when they first breached the data center.
Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)
This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).
Aha, the old replace-your-trusted-hardware trick.
Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this.
Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.
If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive.
It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.
In the git universe there is a pretty short list of services, locally or hosted that you would probably use as an entity as large as cloud flare.
The thing is: What standard of security would you expect of someone who was decrypting 1/3rd of your internet traffic?
I would say "better than most companies" is too low a bar. Hell, I'm not sure any organisation could be secure enough to be trusted with that.
I wouldn't be surprised if they are working on first party IAM user support though.
For example, if Slack devops team were to exclusively communicate over Slack, then a Slack outage would be much harder to resolve because the team trying to fix it would be unable to communicate.
They are using zero trust and explained that it's why the scope of the security incident was extremely limited.
Maybe naively, I wish this assumption became universal.
(Sorry, so sorry, it was just too low-hanging not to to pluck: https://m.youtube.com/watch?v=W2G2sac9s34)
How exactly ?
Nothing out of the ordinary/regular infiltration, investigation and attempt to move laterally is exposed.
thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".
My concern was mostly around the situation where you believe it had not been, but it had.
What I’d expect is a note on the 24th that they’d kicked a threat actor of their Jira system, and that they had no reason to believe the rest of their systems were compromised, but that they were taking action to prevent it from happening again and starting a full investigation.
I get that the business might not want to do that if they are not certain there is any cause for alarm. Uncertainty might be even worse for some.
I'd definitely consider a "silent" credential - a credential not registered centrally - to be a huge red flag. Either it could get stolen, or break and no one knows how to regenerate it. And it's pretty easy as devs to quickly generate an auth key that ends up being used permanently, without any documentation.
Sure, but you aren't going to do all that when your team is juggling N other priorities. At least, it will be very difficult getting mgmt and others on board. Unless it's explicitly in the context of a recent breach.
I don't love CF, but IMO Okta deserves to be punched down on.
Is it really reasonable to come out and say your company utterly failed a pretty basic security practice when faced with a compromise but that it was really some other company's problem originally?
Of course it's not. It's still your company's failure. Own it.
Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?
Reason I ask is long-hosted small business websites are sometimes established with the intent to legitimize some future unrelated traffic.
Probably not what's happening.
I've tried to build a cloud CI service a while ago. Per their nature, you _have to_ allow arbitrary commands to be run. And you also have to allow outbound connectivity. So you don't need to 'own' anything in order to be dangerous. They will not run with heightened privileges but that's of little help if the target is external.
It is pretty difficult to reliably secure them against being used as a source of attacks as there's a lot you can do that will mimic legitimate traffic. Sure, you can block connections to things like IRC and you can throttle or flag some suspicious traffic. You can't really prevent HTTPS requests from going out. Heck, even SSH is pretty much required if you are allowing access to git.
Generally speaking, a build service provider will try to harden their own services and sandbox anything that is run in order to protect themselves from being compromised. Most providers won't want to be known as a major source of malicious activity, so there's some effort there. AWS and other large providers have more resources and will easily ban your ass, but that doesn't matter if it happens after a successful attack was launched.
What the parent is likely referring to is: Things like "Wearing the noogler hat" or the hoops you jump through in interview (that have nothing to do with the job) are similar in spirit to some university fraternities admission processes, or are depicted as such in media.
I guess no need to specific names. I'm just using that as examples.
But everyone here is missing the point of it, it's not to make sure you never get breached it's to ensure forensics exist that cannot be tampered with.
Separation of concerns to keep any single party from within the company from doing anything fraudulent or for an attacker to cover any tracks.
It's not intended at all to be any kind of security by itself outside of the damage an employee can do. Bad code exists and PCI will do nothing to prevent this, because that's not the purpose of the compliance.
IMO it’s more a risk reward trade off. I know some companies are paying relative peanuts in non-compliance fines rather than spend money on some semblance of security which they may still not be compliant with and have to pay the fines anyway…
The whole hosting provider thing was another type of problems. I would always look at who owned the IPs that malicious sign-ups were coming from, and found a lot of ASNs owned by companies like "hosturwebsite4u.or.uk" and things like that. Those I assumed were just forgotten-about Linux boxes that the attackers used to anonymize through.
Ultimately, this was all to get a "free trial" of our cloud service, which did let you run arbitrary code. We eventually had a fairly large number of ASNs that would get a message like "contact sales for a free trial" instead of just auto-approving. That was the end of this particular brand of scammers. (They did contact sales, though! Sales was not convinced they were a legitimate customer, so didn't give them a free trial. Very fun times ;)
I should really write up the whole experience. I learned so much about crypto mining and 2020-era script-kiddie-ing in a very short period of time. My two favorite tangents were 1) I eventually wrote some automation to kill free trials that were using 100% CPU for more than 12 hours or something like that, and so they just made their miner run at 87% CPU. 2) They tried to LD_PRELOAD some code that prevented their process from showing up in the process table, but didn't realize that our tools were statically linked and that they were running in an unprivileged container, so the technique doubly didn't work. But, good old `ps` and `top` are linked against glibc, so they probably fooled a lot of people this way. They also left their code for the libc stub around, and I enjoyed reading it.
The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.
The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.
Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery.
And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.
I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext.
At the very least you need to tell the server what to trust.
That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user <-> Cloudflare <-> origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.
When does the US attack private enterprise?
There's many such cases. They're well known for spying on Siemens as well. With allies like the United States, who needs enemies?
[1] https://www.spiegel.de/politik/ausland/wikileaks-enthuellung...
Also remember the Google sniffing?
https://www.theregister.com/2013/11/07/google_engineers_slam...
The difference really is point in time vs period over time audits. PCI is a point in time audit, SOC 2 is a period over time audit. So for SOC 2 you do need monitoring controls, and then they test that control over the entire period (often 6-12 months). So you are monitoring the control effectiveness over a longer period of time with SOC 2. And even PCI has some period over time controls you need to demonstrate.
From the outside all compliance will seem like checkboxes to most people once controls are established. Because really the goal for most of the business is to make sure the control they interact with doesn't break, and the compliance team will likely give a list of things that the business can't afford to have broken. Which does seem like a checklist similar to PCI. But really, only PCI is straight up a checklist, as you don't really get to decide your controls.
https://twitter.com/taviso/status/1566077115992133634
> True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF?
For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare:
https://twitter.com/taviso/status/832744397800214528
> Could someone from cloudflare security urgently contact me.
This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-c...
Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone who worked as a fairly junior staffer at the FTC. On the employee’s personal time they mentioned being frustrated by how the disclosure took place to the person they were dating. I believe the employee’s frustration was because we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up.
There was never anything that Cloudflare or any executive raised with the FTC. And the FTC never took or even considered taking any action. The junior FTC staffer may have said something to Tavis or our employee may have said something about telling the staffer they were dating, but that was the extent of it.
I understand Tavis’s perspective, and agree it was inappropriate of the former Cloudflare employee, but this was two people not in any position of leadership at either Cloudflare or the FTC talking very much out of school.
This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline.
If you working there and having done research can get it this wrong while it's super easy to find the event log in the open, it doesn't give a very good vibe about the attitude inside cloudflare regarding what happened and fair disclosure.
Full even log on project zero is here : https://bugs.chromium.org/p/project-zero/issues/detail?id=11...
> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
Meanwhile link with Cloudflare went from this
> I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.
> I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.
To this
> Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.
> Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.
> They then told me Wednesday, but in a later reply started saying Thursday.
> I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks. https://security.googleblog.com/2013/05/disclosure-timeline-...
> If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.
> I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.
> They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.
> Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification.
So it was not project zero but cloudflare that moved the disclosure timeline around, and did so without keeping pzero in the loop, about an active in the wild exploit.
> However, Server-Side Excludes are rarely used and only activated for malicious IP addresses.
So… you’re celebrating that you only had buffer overruns for malicious IP addresses?
For context: you are answering to the co-founder & CEO of Cloudflare.
As for covert ops, well, they're covert. I don't have any evidence (hence I said "I'm guessing") but that's how I understand secretive agencies do things. If you look at all of the agencies involved in Stuxnet, you'd get the idea that allied countries' secret services tend to work together (or for each other) to some degree when it suits them.
You think nobody's logged into their personal spotify on their work computer? All those guys wearing headphones in the office have brought in CDs to play in their laptop CD drives?
And that business traveller away from their partner and kids for a week+ isn't going to video call them? Or watch some netflix in their hotel room in the evening?
That's so unrealistic, you could write IT security policy for a Fortune 100 company :)
Why would I use a device to do personal things that they MITM everything I do on it? Privacy is too important to me to give it away like that. I'm sure all traffic on the corporate network is logged. Why open myself up for grounds for termination if my company hits hard times and wants to lay people off?
I've worked for large media companies where this is exactly the only way to have music available. The production network was blocked from accessing the www. To ensure content wasn't pirated, the original media had to be used. No CD-Rs were allowed. Personal devices were kept in lockers outside the restricted areas, so no streaming from them either.
Email was from a remote session. If you were emailed an attachment necessary for production work, there was an approved workflow to scan the data and then make it available to the production network.
So, while you were trying to be sarcastic, there are networks that are set up exactly like you thought didn't exist because it was too outlandish.
I refuse to carry more than one phone or one laptop, and I sure ain’t brining a personal device into a country I wouldnt go to on vacation.
Footgun, but maybe tolerable with your chosen threat model.
Even companies with these policies preinstall Spotify on work computers.
I don't think anyone thinks that. No one also thinks logging into a personal account on a device owned by someone else gives you any claim of ownership over it.
The computer belongs to the company. You will do what the company says you need to with their computer.
Agreed, but I knew many devs in my career who mix personal stuff into work hardware. Maybe its just spotify/pandora, maybe some HR thing they needed their personal gmail to make it easier.
This included "senior" and other levels, it isn't just ppl out of college.
(& then just rotate the credentials on it when you part ways with the employer.)
Some of my co-workers even do a Github account per employment.
I just don’t understand any rational otherwise.
So, would you care to more explicitly tell me what you think about my intelligence or ability to behave rationally compared to you? Or is there potentially some room for nuance here?
A more senior academic?
I'm sure there are plenty of people who have access to their company's private repos through their personal GitHub accounts.
I think this is fairly common for people who work on open source projects.
There was also inferences that they penetrated Huawei.
I might be willing to give you Huawei if you cite a source. They're a gray area (by design) due to China's strategy of military-civil fusion.
Given hacking means unauthorized access to data, can you explain how intercepting confidential documents in an unauthorized manner could not possibly meet the definition?
Additionally, we know much more detail on Siemens, including the planting of malicious code, which absolutely meets any definition of hacking. [1]
- Is the open-source software something that the company is sponsoring?
- If not, do you have permission to use company equipment for personal use?
> A more senior academic?
?
Do you do the above? If so, do you have a personal laptop? if yes, why utilize company property instead of personal, unless given permission to do so?
An example university policy [1]
> 11.5 reasonable personal use of College IT resources is permitted provided such use does not disrupt the conduct of College business or other users. Recreational use of the Halls of Residence network is also permitted, subject to these conditions;
We have a similar policy where I work. I have a personal laptop, but I don't take it to work. I am signed in to my personal GMail account on my work computer, along with many other accounts — like this HN account. If work needed to look at an employee's computer, we'd have someone from IT + someone from HR overseeing the process, and wouldn't look at anything clearly private, e.g. a personal email account. Doing otherwise would be a breach of the GDPR.
[1] https://www.imperial.ac.uk/admin-services/ict/self-service/c...
- people tend to use company devices for private stuff, even when its explicitly prohibited, - draconian policing leads to employee dissatisfaction; you won't be able to fire that great engineer you spent 3 months hiring because he logged in to Spotify running within Chrome, and if you can - and do - soon you will be unable to hire top talent,
Thus, even with those policies in place, end user devices still need to be considered un-trusted. Specifically, that they can be key-logged and remote accessed by the attackers.
Hence, (a) anything sensitive should involve transaction level validation, not just end user authentication, (b) for logging in an out, as well as for confirming sensitive operations, proper MFA needs to be in place (physical key + token on a mobile device, for instance), (c) apply lightweight, reasonable restrictions to reduce the chances of device compromise dramatically (e.g., no downloading of 3rd party apps or binaries - but do whitelist things like Skype or Spotify, force strong password for devices, etc).
This means reasonable personal use is perfectly fine, employees happy, and you are safer vs. assuming local devices are clean.
Been in the industry for a while now, no one cares if you pull out your phone. Generally, people treat others like adults not children.
It was a public case, but the essentially unanimous Supreme Court opinion in City of Ontario v. Quon [0, 2010] shows what expectations of privacy you should have on any work devices -- none.
Reasonable or limited private use of a work computer remains private.
https://edps.europa.eu/data-protection/data-protection/refer...
No thanks. New account per job.
Involvement with private repositories is removed as soon as the organization removes the employee, or the employee removes themselves.
I think the horror stories could only happen if the individual's account has been used for generating many API keys or similar, but there are other reasons not to rely on that sort of thing.
* personal ChatGPT and copilot subscriptions, since company doesn’t pay for these
* Trello account for keeping track of my todo list (following up with people, running deploys)
* Obsidian for keeping notes, as a personal knowledge-base (things like technologies and reminders)
* Apple account for music, copy/paste, sharing photos from my travel with coworkers, synching docs related to my work visa and taxes
* Personal slack login for communicating with my partner in our private server
* personal GitHub account credentials for synching my private dotfiles repo with my neovim config. basically can’t work without my dotfiles, but I could theoretically email these to myself or something, to prevent this one.
And sure, I could be stubborn and not use any of this, but I’d be way less productive and kinda miserable.
* Stack Overflow
* Job Search sites
I don't remember if Jetbrains needs a password to get to personal licenses, but they definitely do to use their bug database. I suspect they're not the only one.
Letting other people blow off steam can be an act of self-preservation. Insisting that people only ever do 100% work things at work or on work hardware slightly raises your low-but-never-zero chances of being murdered by coworkers. Or less ironically, hilariously intense bridge-burning activities.
Also most of this conversation is happening during work hours so I think we can infer that grandparent is being a little hypocritical.
I’m now understanding how people get sued when going from company to company.
> personal Obsidian for keeping meeting notes, and recording conversations as a personal knowledge-base
I'm not a lawyer, but I'm pretty sure these could subject a lot of your other personal data to potential subpoena should your employer get sued by a sufficiently determined attacker.
Don't cross the streams.
> Obsidian is free for personal and non-profit use. However, if you use Obsidian for work-related activities that generate revenue in a company with two or more people, you must purchase a commercial license for each user. Non-profit organizations are exempt from this requirement.
You are making your work take on an extraordinary risk in hiring you.
My notes are text files on the computer, so we’d have problems regardless if they got that. But maybe I should’ve left it out of the list above in that case… nothing else seems very damning.
But you do raise a valid concern, and it’s worth reevaluating!
Does your workplace restrict you from bringing it in?
I’m fine with it because I know there’s no management software on this laptop, but yeah it’s a totally different story if I had to use a newer one with SSO and management software
It’s really easy to say ‘don’t use your personal stuff at work’, but when work is some locked-down behemoth whose view of productivity software is ‘just use Office’, and you’re really trying to be better at your job, using your own tools can be the only solution.
And in my situation, yeah, they didn’t want you bringing things in. I worked in a secure area.
Also, people just do things for convenience. (Although I tend to pipe these passwords over an SSH connection, so that they're not resident on the work laptop. Though there is a good argument to be had about me permitting my work laptop SSH access to my personal laptop. From a technical standpoint, my employer could hack/compromise my personal laptop. From a legal and trust standpoint, I presume they won't.)
You trust all personnel with access to your employers network?
What's more surprising is that they trust you to setup adhoc ssh connections to arbitrary endpoints; unless you're the person in charge of network security?
Would anyone notice if you, or an intruder, dumped terabytes of data over that connection?
I don't work in IT but this just doesn't feel right to me.
Absolutely none of my personal stuff ever touches a corporate machine. Ever. I wouldn't even log in to the W2 downloading app as an employee from the work machine.
Granting work ssh keys access to your personal machine is crazy; if your work machine gets compromised, they steal your entire personal system's home directory too. Why would you unnecessarily expand the blast radius of a compromise like this?
Even if a tiny risk, it seems silly just to bolster the GH activity graph.
> Q3. Can I buy a license for myself, or do I have to ask my company to buy it for me? > Yes, you can buy a license for yourself; just put your name in the company > field. You can use such a license to work for any company.
https://help.obsidian.md/Licenses+and+payment/Commercial+lic...
At least that's how it works in the vast majority of companies.
You wouldn’t keylog “random devs”, you’d keylog all of the ones doing ops.
Finding solutions to work around the systems, on your own time and dime, only hurts in the long run.
They think everything is fine. Nothing will ever get fixed. Voice these concerns.
The tools are the tools. There’s nothing me or my boss or theirs can do about it.
They just don’t care. But I care, because if nothing else it’s my reputation.
(HP used purely for size comparison. I’ve never worked there.)
You might feel a sense of social obligation or solidarity with the company. I usually do. But if I was placed in a dehumanizing situation like that – forced to work inefficiently due to overly rigid policies that assume everyone’s needs are the same – well, whether I worked around it or not, my empathy for the company would be at a nadir whenever I thought about it.
Just because the org that hires you is a shambles doesn’t mean you give up and quit. Thank fuck we don’t all think like that.
And, again, reputation. I have a stellar reputation because I stick it out, and I care. I’ve worked with people who quit because ‘it’s shit here’. Nobody will ever work with them again.
So why raise the point in the first place if it's a minor quibble relative to your top priority(s)?
Tools enable success. Better tools make the job easier. They make the result better.
But having bad tools isn’t a reason to give up. It’s frustrating. But you just have to get on with it.