> In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor.
Like how that amateurish but high profile Wannacry attack revealed a much more lucrative Monero mining botnet that was running with the same exploit for weeks longer, but some script kiddie ruined it
It wasn't one of those "healing viruses" either; it was exploiting a weakness and preventing others from doing the same.
Deterministic builds cannot come soon enough. And really, builds are not enough, we need to be able to extend confidence in the execution of the programs we write much deeper than just builds.
This doesn't do anything for people who buy SolarWinds Orion, which is a closed-source off-the-shelf tool that gets picked up everywhere because of a combination of good sales tactics, compliance checkboxes, and ability to remove work from all involved.
Going back up the chain, a technical solution probably won't solve the issues inside SolarWinds either. Systemic organizational issues lead to RCE backdoors and implants distributed on official update servers, signed with authentic keys.
> The fact that the compromised file is digitally signed suggests the attackers were able to access the company’s software development or distribution pipeline. Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes.
Unless this a compromise of the build machine, it sounds suspiciously like a lack of code review standards to me.
In our organization, the only way to get a line of code into master is through a process where a 2nd developer reviews and approves via GitHub. Branch protection rules are really nice for this kind of concern. Obviously, the attacker can hit right after cloning source, but it helps to know your foundations are clean regardless.
Ken Thompson described this a long time ago: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
At runtime, you alter the build system to notice if a new compiler binary is being built. If so, inject code that, from then on, invisibly "injects the injector". Once you've shifted to this new compiler, the compiler binary itself is the attacker.
Since you have an injector, you can put other types of detections and alterations in...like watching for the compilation of the initialization of a Solarwinds DLL. Then you inject in what you need. No source code is involved.
There are probably lots of ways of getting around this, but unless you're actually looking for it, you won't see it.
The issue is that most developers view "the code pipeline" as a trusted and complete system, for the most part. In the vast majority of cases, that's okay. The issue is that SolarWinds should have known, based on their very own customer list, that they were in an advantageous position in many organizations that are valuable targets. That should have _caused_ all of this thinking to happen, and led to changes internally to accommodate the new risk. That threat modeling/analysis either didn't happen, or the outputs weren't good enough.
> Based on our investigations to date, which are ongoing, we believe that the vulnerability was inserted within the Orion Platform products and existed in updates released between March and June 2020 (what we call the “relevant period”) as a result of a compromise of the Orion software build system and was not present in the source code repository of the Orion Platform products.
Also, it would likely (or hopefully) trigger a static analysis warning in the build as soon as it’s added. For such a sophisticated attack this would be too much of a weak point. It would be much better to have access to a point in the build system that enabled you to inject that code in or after the compilation, e.g by tampering with the tool chain on the build machines.
For what it's worth, my org also has the same policy, but it's intended to catch mistakes, not to protect against malicious actors inside the company.
Imo it’s would be trivial to compromise many. Most companies have soft underbelly units like offshore maintenance engineering, tools teams and patching teams who don’t get a lot of meaningful oversight and can bypass many controls.
Or if Java is a bridge too far, are you inspecting your minified webpack output to ensure no one is inserting malicious Javascript?
Assume compromise of the build machine. Start with, who builds the build machine, and how do they maintain it? Can humans get into it at all, such as in a break glass scenario?
Also, GitHub, GitLab, and the like, may not actually guarantee what you are relying on to enforce “the only way”.
I'd be curious to see if the digitally signed bad versions are similar to digitally signed good versions, i.e. if there's any chance the attacker found/developed a hash collision against an otherwise legitimate build. AIUI it'd be a pretty big deal since it would point to a vulnerability in SHA-256 (which is usually how Windows binaries are signed), but this is apparently a nation state we're dealing with? ¯\_(ツ)_/¯
[1] https://twitter.com/vinodsparrow/status/1338431183588188160/...
You don't need the GRU with such a company. Microsoft Defender would not help. Even a 12 year old from mom's basement could have intruded the nuclear arsenal this way. The nation state allegation came from the stealth CC stuff they found. But apparently someone else also took the invitation via writable ftp.
Just running in the background, waiting to amend a big commit with many changed files/lines would probably go a long way. How often does a reviewer glaze over when reading through a diff where someone shuffled some modules around, causing a lot of line changes without any real implementation changes? Perfect opportunity to slip a few new lines into a long file amidst all the other changes.
No git blame? No bugs around that area? No one questioning what/how/why is that there?
Either that one was used to compromise the supply chain (in which case it makes little to no sense to keep it around and risk detection), or at least 2 different groups had the chance to target sensitive US infrastructure.
Funny how media coverage of this issue misses no chance of mentioning Russia and nobody else, not even possible suspects.
I wonder what happens if the attackers notice each other on the compromised system. Do they get along in exfiltrating data or do they fight quietly?
(Ideally with some automated tooling, too.)
2. Don’t homogenize all the platforms.
3. Reproducible builds; in all likelihood the build server was compromised.
If one of the big 3 superpowers really wants to backdoor your product, then even a top-notch company might fall victim. Hopefully this increased awareness will make it harder to pull off these subtle compromises without it getting caught sooner, though.
> To have some minimal form of obfuscation from prying eyes, the strings in the backdoor are compressed and encoded in Base64, or their hashes are used instead.
There needs to be a quick tool that flags strings that appear to represent binary data before a merge, maybe even decoding them when possible and providing hints of what they might represent, especially inside source-code files. These shouldn't be common in checked code. And we should figure out a way to whitelist them in the repo that's both safe and convenient (I'm not sure how).
Is this a feature code-hosting sites like GitHub can add?
Chuckled at this one.
Is there any further explanation of how this was achieved? One might expect as "par for the course" that all external connections be blocked aside from explicitly designated ranges. I would expect that an attempt at external comms would set off alarms.
"avsvmcloud.com" is far, far more important -- because ALL of the communications go there...
Now, it may be that "avsvmcloud.com" is a legitimate ISP, hosting provider or what-have-you...
But, if I were an investigator on this case, I know I'd want to track each and every place that these requests flow through whoever owns the "avsvmcloud.com" network...
I'd start with the idea that because a subdomain is being used, that the first thing that happens is that subdomains must be resolved by a DNS subdomain servers... so where exactly on whoever owns the "avsvmcloud.com" network, does that happen?
I'd even go so far as to audit, completely dissasemble, the DNS software that is running on those servers... Give it to as many security researchers as possible... What does it do? Where does it point to? What's on the other end of those IP addresses that it resolves to? Are there any anomalies in that IP address resolution? Specifically, when/where and how do they manifest? Are there any patterns there? Who owns the machines on the other side of those IP addresses?
Etc., etc.
In fact...
What would happen if someone were to run a machine learning algorithm on say a, let's be polite and call it a "challenged" DNS resolver?
Would it find some DNS resolution anomalies?
In fact, if I were an investigator, I'd go as far as to audit the whole chain of DNS resolvers / the DNS resolution process THOROUGHLY...
How did we get here?
Some thoughts: if there is an encoded sub domain, flag this as suspicious.
Any code that uses a function to decode a base64 encoded string should be a red flag.
Any newly created thread code should be detected and checked most carefully.
Any others people can think of?
Renting space that shares a netblock with a trusted host could make that more difficult.
I think that the tech industry has a severe code supply chain issue. Supply chains are a super hard problem with physical products (Raise a hand if you (a) have tried tracing the supply chain of cocoa (b) Can tell a midnight factory run of luxury clothes from a legitimate one (c) remember the supermicro controversy ) but with software we have the ability to do a much better job on solving it. I find it really disappointing that we have failed to do so. Reading through the comments here I've seen discussions on deterministic builds, code signing, and other practices. I think that they are parts of a unified whole, but all the pieces need to be there and need to be correctly done. Below I outline where I think the industry should be.
A complete, secure, code supply chain should do the following:
* Validate signatures on all 3rd party dependencies
* Ensure that all internally written code relies on signed commits
* Have builds be reproducible
* Sign the output of those builds
Taken together all of these form a complete supply chain that applies to both closed and open source software. There is nothing technically infeasible about implementing much of it as well - to me it feels like a culture issue.The gap between where we seem to be and where we should be seems to be:
* Validate signatures on all 3rd party dependencies
** Present Day: Many vendors cannot be bothered to properly sign the outputs of their builds. Microsoft updates, openssh releases, and things like that remain the exception rather than the rule. This problem becomes even more egregious when looking at enterprise to enterprise products such as drivers which are either massive sets of source code or precompiled blobs, both of which run with lots of privileges in the context of the product they are integrated into. Even Fedora provides lots of packages from their Koji build system, the majority of which are not signed.
** Where we could be: Normalize signing these, and normalize validating the signatures prior to any use in a build environment. This is one of the easiest places in the supply chain to insert malware due to the lack of verifications.
* Ensure that all internally written code relies on signed commits
** Present Day: Outside of git, most VCS systems don't even support signed commits. Within git, signed commits are not popular. I personally blame the tooling. Signing is based on PGP keys which have all sorts of known issues with use, tooling, and a general disdain due to their initial use case for email being broken. Places like Github attempt features like mandatory signing, but that falls short. Keys are still sourced from unknown places, each developer is responsible for their own key, there is no support for validating prior commits once the signing key is rotated, and using the webui totally bypasses the signing requirements (https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/about-commit-signature-verification).
** Where we could be: Let's imagine a future where git is used as a VCS. Signing keys should be centrally controlled by an authority with developers issued code signing subkeys that are rotated and can be revoked by the central authority. By having a history of all code signing keys over time, the repository can also be audited at any point in time. Even if malicious insiders directly alter the VCS, it can be flagged! I lead a project to implement such a system at my work ( https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/ ) which I am posting on every discussion here to try and normalize a discussion around how to do this at other companies.
* Have builds be reproducible
** Present Day: This is probably the biggest gap in having a secure supply chain. Builds today are not reproducible nor are they deterministic. The best which I know of is NixOS which is around 99% reproducible ( https://r13y.com/ ). Debian appears 95% on a specific target ( https://isdebianreproducibleyet.com/ ). Most other products are much lower than that.
** Where we could be: The first step would be deterministic builds, where building with the same inputs always results in the same outputs. Once you have a way to store what those inputs are, you can then reproduce builds later. Securing build environments becomes much easier at that point. You can build in multiple places, at multiple security levels, and check the same output comes out each time. You can even build at a much later point in time since you should have your whole set of dependencies clearly documented and saved. Validating outputs is super easy later on since you can recreate exactly what it should have been. This is also great for build systems in general since it makes dependency graphs more accurate and reduces problems with building in different environments. With the existence of VMs and containers, this is also a problem that should be super solvable. The devil is in the details here, but there should not be any reason it cannot be solved other than a lack of proper investment.
* Sign the output of those builds
** Where we are today: This is one of the items that is actually the most popular, since it is so easy to do. There are lots of methods to sign any sort of data and the tooling around them is pretty straightforward. By signing this data, it closes the loop on someone downstream validating that data as an input to their own system.
** Where we could be: Keeping up the good work and going further to normalize signing build system outputs!This says to me that the certificate itself probably wasn't compromised. The attacker must have found a place in the CI pipeline where they could insert code and get it signed automatically.
ah so base64 is valid encryption after all.
And finally, it talks about hashes (without even claiming cryptographic hash functions), and but it's not about the base64 strings...
This is also a solution (albeit with a race condition) to one bug that the Morris worm had: the same worm infecting a host multiple times and the multiple instances drawing more attention / stepping on its own feet. Other worms likely have had similar flaws.
(Any suggestions to eloquently un-mix the worm/feet metaphors?)
There are parts of the intelligence community that know with confidence who the true attacker is. Even if they had no idea they were being exploited, there are many ways to perform post-mortem analysis when you're, e.g., the NSA. So, someone has 100% confidence, or close to it.
In terms of what the media says: typically, they report on off-the-record remarks from officials and leaks. That's just how the game is played. It's an unfortunate byproduct of everyone wanting to tell, but nobody wanting to be caught telling. The value of Reuters and AP is that they typically do enough due diligence on their own sources to make sure that they're not just spouting nonsense. "Top of the food chain" sources like them are very regularly correct, but fallible.
This isn't how the game is supposed to be played and is a symptom of the erosion of the media's journalistic integrity. Anonymous sources can tell you where the bodies are buried, but you still need to dig up the bodies. One would think if you're going through all the trouble to track down three different sources who are both competent and trustworthy to comment on who the government suspects, that you'd take the opportunity to ask a follow up question like "why do you think it was them?" Yeah, everyone wants to be the first to break a story, and real investigation is a lot harder than tabloid journalism, but that's the job, or at least that's what it used to be.
> Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes. Therefore, insertion of malicious code into the SolarWinds.Orion.Core.BusinessLayer.dll likely occurred at an early stage, before the final stages of the software build, which would include digitally signing the compiled code.
They talk about adding classes at an early stage... I assumed that meant modifying the source code in some fashion. Sounds like you took it to mean they inserted binary (MSIL?) code during the build? Or are you referring to a different paragraph that indicates this wasn't part of source control?
I'll bet the only result would be confused developers.
[1]: https://docs.github.com/en/free-pro-team@latest/github/admin...
The country is an enigma to me—very good at math but from what I read a corrupt place with rule by power and not law. So yeah, I'm just genuinely asking, how are they the ones doing these things which make the world a better place, and why isn't it us leading the charge in these 3 issues? Maybe it is just a coincidence that those 3 things align with their selfish interests, and their is no altruism involved? Or maybe there's a group in Russia that loves the ideals of the USA, and is able to actually help implement those ideals from there, because if you tried to do those 3 things from here you would be thrown in prison due to some bad laws/people here?
I’ll have to ask then, what proof is there that Russia did this hack? Do you realise how hard it is to track professional hackers? You will have to trace the entire network commands up to a source and hope that it is registered under their name.
I genuinely cannot believe people here think that government managed to find the source of the hack in couple of days, that is just pure propaganda aimed for people with little to no knowledge about computer security.
And granting asylum to persons who have fallen out of favour with an enemy/rival has a been a thing back to at least the time of the Peloponnesian War and probably before.
I think this research was pioneered by US with Stuxnet/Flame? No?
Carefully compiling a corrupted program doesn't fix it.
P.S. Oh, one more thing: some workflows don't mandate a second review after the code is amended, so the initial code could be benign. And many don't require the merger to be the reviewer either. So this could've gotten in that way too.
> small criminal gang
They're the same picture.
Seems to me like this malware might as well have been added to the calling assembly.
An automated tool might have more of a chance, but again it’s kinda hard to have one that runs on binaries. If it runs after build, it’s typically some input/output checker, which would not detect code like this.
It’s a hard problem and I think it has just demonstrated that the security of build-related infrastructure should be taken more seriously than it currently is.
Even if I did have source write access, rather than adding the poison to the runnable code, I’d add the poison to code run at build time (a unit test) which modified the build tool chain and then removed all traces of the poison code again.
That of course leads one to ask what this code is actually doing.
> protecting a whistleblower against government overreach
Becomes protecting an enemy of their enemy, and one that commands a high press influence.
> pointing out how fundamentally insecure closed source, proprietary software
Becomes just exploiting the enemy infrastructure.
The one thing that I'm not comfortable with a simple explanation of "the Russian government doesn't like the US" is their support of SciHub. It can explain the support quite well, but it is not the only simple explanation available, so there may be other reasons.
Has Google relaxed that ban in any way? I.e. does Google believe that Windows 10 is now secure?
What other major corporations or government departments have followed Google’s example since 2010?
Still regarding this specific hack the exploit was hidden in Orion's telemetry, showing that Microsoft's new love of telemetry isn't just privacy invading it's security degrading.
https://www.solarwinds.com/securityadvisory/faq
> Our initial investigations point to an issue in the Orion software build system in which the vulnerability was insert which, if present and activated, could potentially allow an attacker to compromise the server on which the Orion Platform products run.
Under "With these processes in place how was your code compromised?"
1. Source code as available on development machines and version control
2. Source code as available on build machines
3. .dll generated by build machines
I would assume (based on the article's language that you quoted) that manipulation happened on (2), not (1) or (3).
You might be surprised about how even the world's top intelligence agencies sometimes do make simple mistakes with domain and network registration which really are just genuine fuckups rather than false flag subterfuge. This is very rarely a matter of something silly like "Russian IP = Russian intelligence" and more like sloppily re-using an ostensibly non-attributable network or nameserver they didn't realize was already burned.
We're still kind of in the infancy of cyberwarfare. Attribution will probably be harder in a few decades.
But, yes, it's generally a matter of TTPs, target selection, goal analysis, and style.
For anyone curious, they have two excellent articles on this from a few days ago:
https://www.bellingcat.com/resources/2020/12/14/navalny-fsb-...
https://www.bellingcat.com/news/uk-and-europe/2020/12/14/fsb...
There was also an amazing investigation into this published yesterday by a Russian outlet, interviewing some of the black market data brokers and law enforcement officers (both of whom claim some of the brokers will be hunted and killed by the state, now):
https://translate.google.com/translate?sl=auto&tl=en&u=https...
Of course, if following best practices, all build machines should be equally compromised. ;p
Still doesn’t help you if the attack is sufficiently upstream..