Analyzing the compromised DLL file that started the Solorigate attack

Analyzing the compromised DLL file that started the Solorigate attack(microsoft.com)

253 points by thejj100100 5 years ago | 143 comments

eternalny1 5 years ago |

Interesting tidbit at the bottom ...

> In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor.

vmception 5 years ago | |

Is anyone ever pissed that one exploit getting caught reveals other hacker’s efforts?

Like how that amateurish but high profile Wannacry attack revealed a much more lucrative Monero mining botnet that was running with the same exploit for weeks longer, but some script kiddie ruined it

bombcar 5 years ago | | |

I recall at least one "worm" that would patch the hole it came in by, perhaps so that it wouldn't have competition.

It wasn't one of those "healing viruses" either; it was exploiting a weakness and preventing others from doing the same.

infogulch 5 years ago | |

This makes me think of a common refrain when dealing with parasite infestations: If you see one, there's way more than just one.

Deterministic builds cannot come soon enough. And really, builds are not enough, we need to be able to extend confidence in the execution of the programs we write much deeper than just builds.

Shank 5 years ago | | |

> Deterministic builds cannot come soon enough.

This doesn't do anything for people who buy SolarWinds Orion, which is a closed-source off-the-shelf tool that gets picked up everywhere because of a combination of good sales tactics, compliance checkboxes, and ability to remove work from all involved.

Going back up the chain, a technical solution probably won't solve the issues inside SolarWinds either. Systemic organizational issues lead to RCE backdoors and implants distributed on official update servers, signed with authentic keys.

bob1029 5 years ago |

I am curious how this code actually made it in, based upon the following:

> The fact that the compromised file is digitally signed suggests the attackers were able to access the company’s software development or distribution pipeline. Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes.

Unless this a compromise of the build machine, it sounds suspiciously like a lack of code review standards to me.

In our organization, the only way to get a line of code into master is through a process where a 2nd developer reviews and approves via GitHub. Branch protection rules are really nice for this kind of concern. Obviously, the attacker can hit right after cloning source, but it helps to know your foundations are clean regardless.

rossjudson 5 years ago | |

All you need is to be able to influence the behavior of the build system at runtime. If you have that, you do not need access to the source code, and you do not need to check anything in. This includes scenarios where the tool chain itself is checked into the source control system.

Ken Thompson described this a long time ago: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

At runtime, you alter the build system to notice if a new compiler binary is being built. If so, inject code that, from then on, invisibly "injects the injector". Once you've shifted to this new compiler, the compiler binary itself is the attacker.

Since you have an injector, you can put other types of detections and alterations in...like watching for the compilation of the initialization of a Solarwinds DLL. Then you inject in what you need. No source code is involved.

There are probably lots of ways of getting around this, but unless you're actually looking for it, you won't see it.

saagarjha 5 years ago | | |

Successfully pulling off such an attack, of course, is quite difficult.

whoisthemachine 5 years ago | | |

This is really interesting - really seems to be an argument in favor of all code being open, shared source - along with deterministic builds, it would be easy enough for any organization to build Orion themselves and verify they get the correct build hash. Or if there was a disagreement in hashes between SolarWinds and their imagined community, it would serve as a red flag.

Shank 5 years ago | |

If you view GitHub as a static infallible source, then yes -- your analysis is correct. But there are gaps anywhere. If you want perfect 1:1 mappings between source, the developers who make it, and the end builds, you essentially need a "chain of trust" that can be tested at every stage. For example: are all developers pushing code with encrypted SSH keys? Are the commits signed? Are the signing keys hardware backed? Does CI check the signatures? Is the CI server up to date? Are all packages on the CI server signed and trusted? Are all stages of the build pipeline testable for tampering and tamper-proof? You're not curling or apt-getting or running npm anywhere in your build server for some kind of Slack integration, right? The list goes on.

The issue is that most developers view "the code pipeline" as a trusted and complete system, for the most part. In the vast majority of cases, that's okay. The issue is that SolarWinds should have known, based on their very own customer list, that they were in an advantageous position in many organizations that are valuable targets. That should have _caused_ all of this thinking to happen, and led to changes internally to accommodate the new risk. That threat modeling/analysis either didn't happen, or the outputs weren't good enough.

blastro 5 years ago | | |

very well articulated, thanks.

eternalny1 5 years ago | |

They are claiming that their build system was compromised and the code was not under source control.

> Based on our investigations to date, which are ongoing, we believe that the vulnerability was inserted within the Orion Platform products and existed in updates released between March and June 2020 (what we call the “relevant period”) as a result of a compromise of the Orion software build system and was not present in the source code repository of the Orion Platform products.

ashleyn 5 years ago | | |

In my experience and where I work, the build system tends to be the most neglected part of the pipeline, most trouble-prone and frequently the source of headaches nobody wants to bother with. I think the days of build being the red-headed stepchild nobody wants to deal with is coming to an abrupt end.

dboreham 5 years ago | | |

Yarn cache? (or similar bogus feature in other language tooling).

nwellinghoff 5 years ago | |

Spoken like a true dev There are other ways of checking code in besides the official channel. Almost every company on the planet could fall victim to this type of attack. Once a team gets past a certain size and “its not my job” comes into play. All kinds of doors swing open.

alkonaut 5 years ago | | |

It’s too obvious even for a huge app. The empty catch alone is something I’d immediately “git blame” if I saw it. I work on a 20 year old massive enterprise app and there is lots of “not my job”, but someone would see it.

Also, it would likely (or hopefully) trigger a static analysis warning in the build as soon as it’s added. For such a sophisticated attack this would be too much of a weak point. It would be much better to have access to a point in the build system that enabled you to inject that code in or after the compilation, e.g by tampering with the tool chain on the build machines.

tsimionescu 5 years ago | |

If we're at the level where we think it's an inside job, it doesn't seem that difficult to have 2 people on the inside "reviewing" each other's malicious commits.

For what it's worth, my org also has the same policy, but it's intended to catch mistakes, not to protect against malicious actors inside the company.

Spooky23 5 years ago | | |

The vast majority of software shops don’t even consider insider threat in any meaningful way.

Imo it’s would be trivial to compromise many. Most companies have soft underbelly units like offshore maintenance engineering, tools teams and patching teams who don’t get a lot of meaningful oversight and can bypass many controls.

phendrenad2 5 years ago | |

Do you code review the output of your build system? Imagine you had to write Java for a minute. Are you going to open up JAR files and look at strings to ensure no one is inserting code from your build server?

Or if Java is a bridge too far, are you inspecting your minified webpack output to ensure no one is inserting malicious Javascript?

Terretta 5 years ago | |

> Unless this a compromise of the build machine ...

Assume compromise of the build machine. Start with, who builds the build machine, and how do they maintain it? Can humans get into it at all, such as in a break glass scenario?

Also, GitHub, GitLab, and the like, may not actually guarantee what you are relying on to enforce “the only way”.

newhouseb 5 years ago | |

Solarwinds accidentally leaked (via Github) the FTP credentials to the infrastructure used to distribute builds in late 2019 [1].

I'd be curious to see if the digitally signed bad versions are similar to digitally signed good versions, i.e. if there's any chance the attacker found/developed a hash collision against an otherwise legitimate build. AIUI it'd be a pretty big deal since it would point to a vulnerability in SHA-256 (which is usually how Windows binaries are signed), but this is apparently a nation state we're dealing with? ¯\_(ツ)_/¯

[1] https://twitter.com/vinodsparrow/status/1338431183588188160/...

rurban 5 years ago | | |

They did sign it with the key they found there, virus vendors detected fancy bear or such, customer support was in denial and recommended all customer to ignore this warning and disable scanning this binary, whitelist.

You don't need the GRU with such a company. Microsoft Defender would not help. Even a 12 year old from mom's basement could have intruded the nuclear arsenal this way. The nation state allegation came from the stealth CC stuff they found. But apparently someone else also took the invitation via writable ftp.

mekkkkkk 5 years ago | |

Branch protection settings are also editable by someone. What's to say the attacker couldn't disable it or bypass it for this one commit? Also I don't think it's uncommon to have a way to get a hotfix deployed without going through the normal checks and balances. For those "the service is down" calls at 2 AM.

maccard 5 years ago | |

I've said this before, but I work on a team of 5 or 6 people. If I (pre covid) sent them a PR and walked over to their desk, told them it was super urgent and a tiny change just needed a rubber stamp, one of them would do it (and I would likely do the same for them). Failing that I can name a handful of developers that wouldn't be familiar with the system but will review my change because I did the same for them a few months ago (and they'll comment on stylistic/clarity issues, rather than the work being done). Even if you think this is rare, it likely isn't and likely happens at every company to some degree.

yenwel 5 years ago | | |

I've worked at a mediacorp in user authentication team where one rogue junior developer from another team (with the most seniority though at that company subdividion because everybody else left) went behind my back to pressure my junior colleague to merge a pr in our codebase which opened a security hole in the back end because he was working together with a project lead who promised to deliver something that we couldn't.

macNchz 5 years ago | |

Code review and protected master are certainly important but not infallible. If I were a malware author with code running on an owned dev machine and my goal was to sneak code into a repo, I can think of a bunch of strategies that might increase the odds of slipping past a review.

Just running in the background, waiting to amend a big commit with many changed files/lines would probably go a long way. How often does a reviewer glaze over when reading through a diff where someone shuffled some modules around, causing a lot of line changes without any real implementation changes? Perfect opportunity to slip a few new lines into a long file amidst all the other changes.

netsharc 5 years ago | | |

As many are saying here, why commit the malware, just modify Jenkins to detect that it's compiling this DLL and add those lines into the source about to be compiled..

dboreham 5 years ago | |

I offer up a glass of kool aid if you believe this works. Employee A is just going to DM slack Employee B asking them to approve their 7000 like PR because they're going on vacation next week and Employee C has been slow reviewing.

raverbashing 5 years ago | |

I'd be even more curious to know why people saw that code and thought nothing of it.

No git blame? No bugs around that area? No one questioning what/how/why is that there?

codenesium 5 years ago | | |

They likely put a program on the build server to make the changes after checkout before build. Which would be impossible to detect.

mehrdadn 5 years ago | |

It's not hard to envision code reviews that don't review every single line. Been on both sides of it. Code review isn't a security barrier, it's a (noisy) safety check. It can't even catch every silly bug, let alone deliberate covert sabotage.

sorokod 5 years ago | | |

The chunk of code that unzips the Base64 encoded strings looks super dodgy and should jump right out even at a casual reviewer.

chris_wot 5 years ago | |

Can’t you just run a git rebase to disguise it?

saagarjha 5 years ago | | |

Depending on what you rebase to and what the usual workflow is for developers. Those that rebase from remote frequently (guilty) won’t notice, but those that do merges might.

smspf 5 years ago |

>In an interesting turn of events, the investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor.

Either that one was used to compromise the supply chain (in which case it makes little to no sense to keep it around and risk detection), or at least 2 different groups had the chance to target sensitive US infrastructure.

Funny how media coverage of this issue misses no chance of mentioning Russia and nobody else, not even possible suspects.

I wonder what happens if the attackers notice each other on the compromised system. Do they get along in exfiltrating data or do they fight quietly?

otterley 5 years ago |

I would very much like to see prevention advice tacked on to analyses like these. It's very interesting to see how the vulnerabilities were exploited, but I think it would be extremely valuable to understand how to prevent future attacks such as this. What were the root causes of the vulnerability, and how can the community prevent similar ones from being created in the future?

(Ideally with some automated tooling, too.)

mrtesthah 5 years ago | |

1. Don’t outsource 18,000 organizations’ security to the same vendor.

2. Don’t homogenize all the platforms.

3. Reproducible builds; in all likelihood the build server was compromised.

frongpik 5 years ago | | |

I like comparing this type of unification with margin trading, in this case with 18000:1 leverage: if stocks go up, you get many multiples of the usual profits, but it takes only a tiny dip in the price to wipe out the account balance.

jeffbee 5 years ago | |

The root cause? It's that people buy snake oil from vendors to check boxes on meaningless compliance tests.

meowface 5 years ago | | |

It can affect non-snake oil from good vendors who provide useful solutions for meaningful compliance tests, too, though. Or just any popular B2B software provider.

If one of the big 3 superpowers really wants to backdoor your product, then even a top-notch company might fall victim. Hopefully this increased awareness will make it harder to pull off these subtle compromises without it getting caught sooner, though.

dboreham 5 years ago | |

Quick off the top of my head advice is to build on ephemeral systems a la GH Actions / Circle CI, not on long-lived build machines a la Jenkins.

mehrdadn 5 years ago |

Any GitHub/GitLab/etc. employees here? I think you might be able to help mitigate some of these kinds of attacks:

> To have some minimal form of obfuscation from prying eyes, the strings in the backdoor are compressed and encoded in Base64, or their hashes are used instead.

There needs to be a quick tool that flags strings that appear to represent binary data before a merge, maybe even decoding them when possible and providing hints of what they might represent, especially inside source-code files. These shouldn't be common in checked code. And we should figure out a way to whitelist them in the repo that's both safe and convenient (I'm not sure how).

Is this a feature code-hosting sites like GitHub can add?

bassman9000 5 years ago |

This method is part of a class, which the attackers named OrionImprovementBusinessLayer to blend in with the rest of the code

Chuckled at this one.

richardjennings 5 years ago |

"Finally, the backdoor composes a JSON document into which it adds the unique user ID described earlier, a session ID, and a set of other non-relevant data fields. It then sends this JSON document to the C2 server."

Is there any further explanation of how this was achieved? One might expect as "par for the course" that all external connections be blocked aside from explicitly designated ranges. I would expect that an attempt at external comms would set off alarms.

nhoughto 5 years ago | |

I was wondering the same, if the compromise is of the Orion product which presumably isn't just sitting there with open access to the internet? Like this doesn't seem to be a very broad exploit that could start on a machine with outside world access (like a Windows exploit itself) and then pivot into more sensitive areas.

Corrado 5 years ago | |

It's my understanding that multiple, unexplained NXDOMAIN responses IS what exposed the compromised systems. Why it didn't happen earlier (or immediately) is a good question.

peter_d_sherman 5 years ago |

Observation: "avsvmcloud.com" -- seems to be the one constant around which a whole bunch of other things, which are variables revolve... (oh sure, "appsync-api" also appears to be a constant -- but it exists at a far less important place in the URL).

"avsvmcloud.com" is far, far more important -- because ALL of the communications go there...

Now, it may be that "avsvmcloud.com" is a legitimate ISP, hosting provider or what-have-you...

But, if I were an investigator on this case, I know I'd want to track each and every place that these requests flow through whoever owns the "avsvmcloud.com" network...

I'd start with the idea that because a subdomain is being used, that the first thing that happens is that subdomains must be resolved by a DNS subdomain servers... so where exactly on whoever owns the "avsvmcloud.com" network, does that happen?

I'd even go so far as to audit, completely dissasemble, the DNS software that is running on those servers... Give it to as many security researchers as possible... What does it do? Where does it point to? What's on the other end of those IP addresses that it resolves to? Are there any anomalies in that IP address resolution? Specifically, when/where and how do they manifest? Are there any patterns there? Who owns the machines on the other side of those IP addresses?

Etc., etc.

In fact...

What would happen if someone were to run a machine learning algorithm on say a, let's be polite and call it a "challenged" DNS resolver?

Would it find some DNS resolution anomalies?

In fact, if I were an investigator, I'd go as far as to audit the whole chain of DNS resolvers / the DNS resolution process THOROUGHLY...

breck 5 years ago |

Anyone else find it ironic that the country that is responsible for democratizing access to scientific research (via support of SciHub), the country protecting a whistleblower against government overreach (Snowden), the country pointing out how fundamentally insecure closed source, proprietary software is (SolarWinds), is...Russia?

How did we get here?

WarOnPrivacy 5 years ago |

The handling of this whole event is in stark contrast to just a few years ago - when details on malicious activity were a closely guarded secret and useful threat data was safely locked away from anyone who might protect the public with it.

jijji 5 years ago |

I wonder how much different this would be had it been a linux application running under apparmor or in a container environment... One would expect from a security perspective that all of these remotely distributed applications would be running under some kind of chroot jail or container to prevent the kind of exposure that is obviously happening here. I think Microsoft is a little complicit in their lack of security in their OS platform allowing these types of issues to proliferate repeatedly year after year with no real changes happening in the ways that applications are locked down.

dboreham 5 years ago | |

Isn't the whole point to this that the targeted software is supposed to run at high privileges and is also supposed to phone home? So it's the ideal vector for an exfil attack. The only way to avoid it would be to do like Hillary and run your own email server with none of this cool stuff installed.

chris_wot 5 years ago |

I am curious what network detection could be done to detect this sort of thing. Clearly the code needed to make outbound connections to hosts.

Some thoughts: if there is an encoded sub domain, flag this as suspicious.

Any code that uses a function to decode a base64 encoded string should be a red flag.

Any newly created thread code should be detected and checked most carefully.

Any others people can think of?

WarOnPrivacy 5 years ago | |

> what network detection could be done to detect this sort of thing. Clearly the code needed to make outbound connections to hosts.

Renting space that shares a netblock with a trusted host could make that more difficult.

allyant 5 years ago | |

Any decent static code analysers should be able to detect things like this (catch all’s statements, base64 encoding etc), I am surprised none seem to be used for production code.

derwiki 5 years ago | | |

Is SCA often set up to run on the fully built end result running on the production machine? I’ve generally seen them as pre-merge-to-source-control.

er4hn 5 years ago |

NOTE: The views I'm expressing here are solely my own and say nothing about my employers views.

I think that the tech industry has a severe code supply chain issue. Supply chains are a super hard problem with physical products (Raise a hand if you (a) have tried tracing the supply chain of cocoa (b) Can tell a midnight factory run of luxury clothes from a legitimate one (c) remember the supermicro controversy ) but with software we have the ability to do a much better job on solving it. I find it really disappointing that we have failed to do so. Reading through the comments here I've seen discussions on deterministic builds, code signing, and other practices. I think that they are parts of a unified whole, but all the pieces need to be there and need to be correctly done. Below I outline where I think the industry should be.

A complete, secure, code supply chain should do the following:

  * Validate signatures on all 3rd party dependencies
  * Ensure that all internally written code relies on signed commits
  * Have builds be reproducible
  * Sign the output of those builds

Taken together all of these form a complete supply chain that applies to both closed and open source software. There is nothing technically infeasible about implementing much of it as well - to me it feels like a culture issue.

The gap between where we seem to be and where we should be seems to be:

  * Validate signatures on all 3rd party dependencies
  ** Present Day: Many vendors cannot be bothered to properly sign the outputs of their builds. Microsoft updates, openssh releases, and things like that remain the exception rather than the rule. This problem becomes even more egregious when looking at enterprise to enterprise products such as drivers which are either massive sets of source code or precompiled blobs, both of which run with lots of privileges in the context of the product they are integrated into. Even Fedora provides lots of packages from their Koji build system, the majority of which are not signed.
  ** Where we could be: Normalize signing these, and normalize validating the signatures prior to any use in a build environment. This is one of the easiest places in the supply chain to insert malware due to the lack of verifications.
  * Ensure that all internally written code relies on signed commits
  ** Present Day: Outside of git, most VCS systems don't even support signed commits. Within git, signed commits are not popular. I personally blame the tooling. Signing is based on PGP keys which have all sorts of known issues with use, tooling, and a general disdain due to their initial use case for email being broken. Places like Github attempt features like mandatory signing, but that falls short. Keys are still sourced from unknown places, each developer is responsible for their own key, there is no support for validating prior commits once the signing key is rotated, and using the webui totally bypasses the signing requirements (https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/about-commit-signature-verification).
  ** Where we could be: Let's imagine a future where git is used as a VCS. Signing keys should be centrally controlled by an authority with developers issued code signing subkeys that are rotated and can be revoked by the central authority. By having a history of all code signing keys over time, the repository can also be audited at any point in time. Even if malicious insiders directly alter the VCS, it can be flagged! I lead a project to implement such a system at my work ( https://eos.arista.com/commit-signing-with-git-at-enterprise-scale/ ) which I am posting on every discussion here to try and normalize a discussion around how to do this at other companies.
  * Have builds be reproducible
  ** Present Day: This is probably the biggest gap in having a secure supply chain. Builds today are not reproducible nor are they deterministic. The best which I know of is NixOS which is around 99% reproducible ( https://r13y.com/ ). Debian appears 95% on a specific target ( https://isdebianreproducibleyet.com/ ). Most other products are much lower than that.
  ** Where we could be: The first step would be deterministic builds, where building with the same inputs always results in the same outputs. Once you have a way to store what those inputs are, you can then reproduce builds later. Securing build environments becomes much easier at that point. You can build in multiple places, at multiple security levels, and check the same output comes out each time. You can even build at a much later point in time since you should have your whole set of dependencies clearly documented and saved. Validating outputs is super easy later on since you can recreate exactly what it should have been. This is also great for build systems in general since it makes dependency graphs more accurate and reduces problems with building in different environments. With the existence of VMs and containers, this is also a problem that should be super solvable. The devil is in the details here, but there should not be any reason it cannot be solved other than a lack of proper investment.
  * Sign the output of those builds
  ** Where we are today: This is one of the items that is actually the most popular, since it is so easy to do. There are lots of methods to sign any sort of data and the tooling around them is pretty straightforward. By signing this data, it closes the loop on someone downstream validating that data as an input to their own system.
  ** Where we could be: Keeping up the good work and going further to normalize signing build system outputs!

danjc 5 years ago |

I’d have expected that a catch block with nothing in it would have warranted some investigation

metta2uall 5 years ago |

I wonder if there will be a "post-mortem" regarding why Microsoft didn't detect this attack earlier..? (similar question for other security vendors)

MarekKnapek 5 years ago |

Did SolarWinds' CA (certificate authority) reworked their code signing certificate?

EvanAnderson 5 years ago | |

I do work for a SolarWinds Customer. SolarWinds told us on Thursday that the certificate was going to be revoked on the 21st. Then yesterday they told us the certificate wasn't being revoked until February 2021.

This says to me that the certificate itself probably wasn't compromised. The attacker must have found a place in the CI pipeline where they could insert code and get it signed automatically.

dboreham 5 years ago | | |

I'd be surprised if signing was done automatically, that would be really bad. More likely it was done manually on a package that came out their build system, without anyone stroking their beard to wonder if that system had had its compiler replaced or its cache of dependencies poisoned.

tester756 5 years ago |

What's the thing about adding "-gate" to everything?

WarOnPrivacy 5 years ago | |

(not an answer but) It's less annoying than needlessly adding "Cyber" to criminal activity.

krapp 5 years ago | |

it's been a common way to refer to a scandal since Watergate[0]. Don't know if that's actually the case here though.

[0]https://en.wikipedia.org/wiki/Watergate_scandal

jjk166 5 years ago | | |

I wouldn't refer to getting hacked as a scandal. Unless it comes out that there was some sort of coverup, it seems innapropriate to refer to it as a -gate.

tester756 5 years ago | | |

but why? why cannot it just be "solarwinds hack"

mad_vill 5 years ago |

> To have some minimal form of obfuscation from prying eyes, the strings in the backdoor are compressed and encoded in Base64, or their hashes are used instead.

ah so base64 is valid encryption after all.

yuribro 5 years ago | |

It says encoded, not encrypted. base64 is an encoding. It also says obfuscation, which isn't encryption either.

And finally, it talks about hashes (without even claiming cryptographic hash functions), and but it's not about the base64 strings...