https://www.openwall.com/lists/oss-security/2024/03/29/4

[1] https://github.com/JiaT75?tab=following

https://ubuntu.com/security/notices/USN-5378-2

This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt.

The fees would be paid to the service providers as mining fees.

E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.

I ran "brew upgrade" and that downgraded to version 5.4.6.

https://www.nongnu.org/lzip/

https://www.nongnu.org/lzip/xz_inadequate.html

I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)

The real villain is not the perpetrator, it is Microsoft, and it is all of us.

I'm recalling bad memories of the Juniper backdoor years ago.

Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.

I agree with the lzip guy

https://www.nongnu.org/lzip/xz_inadequate.html

if [ "$path" == "" ]

should be

if [ "$path" = "" ]

[0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...

[1] https://github.com/python/cpython-source-deps/tree/xz

5.4.5 can be compromised

https://github.com/microsoft/vcpkg/issues/37197

The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.

See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community.

It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.

Github just disabled the repo : https://github.com/tukaani-project/xz

Do someone have an up to date fork to see the project history ?

For every one of these we spot, assume there are two we have not.

It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.

https://xkcd.com/2347/

nice

IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure.

I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).

These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:

3: + 0200 (in winter: February and November)

6: +0300 (in summer: in Jun, Jul, early October)

440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.

2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March).

Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800

Jia Tan Sat, 19 Nov 2022 23:18:04 +0800

Jia Tan Mon, 7 Nov 2022 16:24:14 +0200

Jia Tan Sun, 23 Oct 2022 21:01:08 +0800

Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)

Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)

Jia Tan Wed, 5 Oct 2022 23:54:12 +0800

Jia Tan Wed, 5 Oct 2022 20:57:16 +0800

—> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800

Jia Tan Thu, 8 Sep 2022 15:07:00 +0300

Jia Tan Mon, 25 Jul 2022 18:30:05 +0300

Jia Tan Mon, 25 Jul 2022 18:20:01 +0300

Jia Tan Fri, 1 Jul 2022 21:19:26 +0800

Jia Tan Thu, 16 Jun 2022 17:32:19 +0300

Jia Tan Mon, 13 Jun 2022 20:27:03 +0800

—> the ordering of these time stamps, and the switching back and forth looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800

Jia Tan Thu, 15 Feb 2024 01:53:40 +0800

Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

Jia Tan Tue, 13 Feb 2024 22:38:58 +0800

—> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023):

- Working on 2023, 29 September: Mid Autumn Festival

- Working on 2023, 05 April: Tomb Sweeping Day

- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays:

- Never working on Dec 25: Christmas (for many EET countries)

- Never working Dec 31 or Jan 1: New Years

Observation 5: No weekend work —> salary job?

The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.

The purpose of this commentary is focused on open source information (OSINT).

I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred.

NOTE: This comment will always have "edit" so always review it for information.

Information I have so far.

Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities.

2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems.

3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes.

4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues.

5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia.

6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities.

Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760

Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...

A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps

It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years

Observation 5: Little weekend work —> salary job?

The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

At the time I thought it was just rude, but maybe this is when it all started.

Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.

EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.

https://packages.ubuntu.com/mantic/liblzma5

And in unreleased 24.04 is 5.4.5-0.3

https://packages.ubuntu.com/noble/liblzma5

There are no changelog entries indicating that the package was reverted.

If the source is included in those images, we could conceivably prove that the target was based on the source.

It's not nice and easy, true.

"However, building from source wouldn’t have protected you against this specific backdoor." Depends on how exactly you build from source. A generic build was not the target. Andres Freund showed that the attack was targeted against a specific type of build system.

It's a known fact that China will "recruit" people to operate them. A quote:

> They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts...

This just gives feelings of less "compromised account" and more "Your account is now our account"

WHOWAS jiatan provided me the following information:

jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024

WHOIS yields nothing, the user is not present on the network at the moment.

Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.

zero definition of what that means...

egos of people who just like to say cool words they don't understand

lol

this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad

state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on.

i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid

Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.

Governments are well known to keep vulnerabilities hidden (see EternalBlue). Intentionally introducing a vulnerability doesn’t seem that backwards tbh

But, anyway, I'm sure we can find other counter-examples.

> Versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key . The older releases have been signed with Lasse Collin's OpenPGP key .

It must be assume that before acquiring that privilege, they also contributed code to project. Probably most was to establish respectable record. Still could be malicious code going back someways.

- All the posts are from 2004/2006. - "jiat" can be abbreviation for many common Chinese names.

In fact, that'd be the best form of deep cover. It'll be interested to watch as people more knowledgable than I pour over every single commit and change.

these files are also useful to check that the library we just built works correctly. but they aren't necessary for installation.

we may have more sophisticated procedures that will allow us to use some parts of distribution only for tests. This may significantly reduce an attack vector - many projects have huge, sophisticated testing infrastructure where you can hide the entire Wikipedia.

That, and I used to contribute to various games (forks of ioquake3) when I was a teen and I wanted to keep my real name private.

That's just what someone with a false identity would say.. get him boys!

The biggest /S

But in this case we are talking about people (distro packagers) manually downloading the source and building it which is not quite the same thing.

This is from Gnulib, which is used by Gettext and other GNU projects. Using 'setlocale (0, NULL)' is not thread-safe on all platforms. Gnulib has modules to work around this, but not all projects want the extra locking. Hence the name '-unsafe'. :)

See: https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg001...

    $ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1

Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)

Obviously a bad actor will make use of these conditions and the assumption of good will.

We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.

it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.

The known unknowns can be better than the unknown unknowns.

My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected.

Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.

Could have fooled me - impressive write-up!

It is too easy to hide things in testdata.

https://web.archive.org/web/20240329182300/https://www.openw...

[1] https://fulda.social/@Ganneff/112184975950858403

Should we start doing background checks on all committers to such critical IT infrastructure?

Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?

I can't track them and I don't want them in my projects.

Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.

It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

Ubuntu 22.04 version:

dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library

Whew!

This is absolutely not the case in many parts of the world.

I agree on the sshd linking part.

https://archlinux.org/news/the-xz-package-has-been-backdoore...

as to the specific comment:

> Seems like the complexity of XZ has backfired severely, as expected.

to summarise: someone found a project with a vulnerable maintenance situation, spent years getting involved in a project, then got commit rights, and then commited a backdoor in some binaries and the build system, then got sock puppets to agitate for OSes to adopt the backdoored code.

the comment I replied to made a "shallow" claim of complexity without any details, so let's look at some possible interpretations:

- code complexity - doesn't seem super relevant - the attacker hide a highly obfuscated backdoor in a binary test file and committed it - approximately no one is ever going to catch such things without a process step of requiring binaries be generatable in a reasonable-looking and hopefully-hard-to-backdoor kind of way. cryptographers are good at this: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number

- build complexity - sure, but it's auto*, that's very common.

- organisational complexity - the opposite is the case. it had one guy maintaining it, who asked for help.

- data/file format complexity - doesn't seem relevant unless it turns out the obfuscation method used was particularly easy for this format, but even in that case, you'd think others would be vulnerable to something equivalent

perhaps OP had some other thing in mind, but then they could have said that, instead of making a crappy comment.

It is kinda sus when they do it at home without consent.

1) and 3) (and, to an extent, 2) )are routinely done, to some degree, by your average security-conscious employer. Your employer knows who you are and probably put some thought on how to avoid your accounts getting hacked.

But what is reliability? Could be anything from "this dude has no outstanding warrants" to "this dude has been extensively investigated by a law enforcement agency with enough resources to dig into their life, finances, friends and family, habits, and so on".

I might be willing to go through these hoops for an actual, "real world" job, but submitting myself to months of investigation just to be able to commit into a Github repository seems excessive.

Also, people change, and you should be able to keep track of everyone all the time, in case someone gets blackmailed or otherwise persuaded to do bad things. And what happens if you find out someone is a double agent? Rolling back years of commits can be incredibly hard.

It is a little different but a thing that you might have missed in the quick read is that one of the things he was accused of was installing and using a backdoor.

Note that I'm not looking for the vulnerable symbols, I'm looking for the library that does the patching in the first place.

You say that as if members of US government agencies didn't plot terror attacks on Americans (Operation Northwood), steal the medical records of American whistleblowers (Ellsberg), had to be prevented from assassinating American journalists (Gordon Liddy, on Jack Anderson), collude to assassinate American political activists (Fred Hampton), spy on presidential candidates (Watergate), sell weapons to countries who'd allegedly supported groups who'd launched suicide bombing attacks on American soldiers (Iran-Contra), allow drug smugglers to flood the USA with cocaine so that they could supply illegal guns to terrorists abroad on their return trip (Iran-Contra again) and get caught conducting illegal mass-surveillance on American people as a whole (Snowden). Among others.

It's super-naive to suggest that government agencies wouldn't act against the interest of American citizens and companies because there might be consequences if they were caught. Most of the instances above actually were instances where the perpetrators did get caught, which is why we know about them.

I think the best reason to doubt USG involvement is the ease with which somebody discovered this issue, which is only a month or two old. I feel like NSA etc. knows not to get caught doing this so easily.

And that is exactly why backdoored encryption is bad.

I’ve been a package maintainer for a decade. I make it a habit to spot check the source code of every update of every upstream package, hoping that if many others do the same, it might make a difference.

But this backdoor? I wouldn’t have been able to spot it to save my life.

At the end of the day we have to be able to answer why this happened, and how we can prevent it from happening again. It's not about pointing fingers, but about improving the process.

BTW, there have been several attempts at introducing backdoors in the Linux kernel. Some manage to go through, and perhaps we don't know about others, but many were thwarted due to the extreme vigilance of maintainers. Thankfully so, as everyone is well aware of how critical the project is. I'm not saying that all projects have the resources and visibility of Linux, but clearly vigilance is a requirement for lowering the chances of this happening.

> That's assuming a maintainer wasn't compromised, like in this case.

What makes you say that? Everything I've read about this (e.g. [1]) suggests that this was done by someone who also made valid contributions and gained gradual control of the project, where they were allowed to bypass any checks, if they existed at all. The misplaced trust in external contributions, and the lack of a proper peer review process are precisely what allowed this to happen.

[1]: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

That's devastating.

If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Not sure what to make of this.

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Removes instructions about details relevant to security reports. Heh, nice one.

https://github.com/tukaani-project/xz vs https://github.com/JiaT75

I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.

https://aws.amazon.com/security/security-bulletins/AWS-2024-...

Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?

A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.

I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad":

I used this regex to split in python, and wrote to "tmp":

    re.split(b'\xfd7zXZ', x)

    $ ent tmp2 # bad file payload
    Entropy = 4.157806 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 48 percent.
    
    Chi square distribution for 48 samples is 1114.67, and randomly
    would exceed this value less than 0.01 percent of the times.
    
    Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
    Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
    Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
    
    $ ent tmp3 # urandom
    Entropy = 5.376629 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 32 percent.
    
    Chi square distribution for 48 samples is 261.33, and randomly
    would exceed this value 37.92 percent of the times.
    
    Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
    Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
    Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).
    

``` #!/bin/bash

# Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")

# Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")

    # Execute xz --version inside the container
    version=$(docker exec "$container" xz --version)

    # Write container name, image, and command output to a text file
    echo "Container: $container" >> docker_container_versions.txt
    echo "Image: $image" >> docker_container_versions.txt
    echo "xz Version:" >> docker_container_versions.txt
    echo "$version" >> docker_container_versions.txt
    echo "" >> docker_container_versions.txt

echo "Output written to docker_container_versions.txt" ```

Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc.

But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands".

The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive)

So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.

I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.

[0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests

I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…

But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.

It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

https://github.com/emirkmo/xz-backdoor-github

For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)

This is a tragedy of the commons, and we can't place blame on a single project besides xz itself, yet we can all share part of the blame to collectively do better in the future.

twitter.com/stskeeps/status/1774019709739872599

On the flip side the elegant/readable build system means that the place this exploit was hidden wouldn't exist. Though I wouldn't confidently say that 'no hiding places exist' (especially with the parts of the ecosystem that wrap dependencies in other languages).

https://github.com/tukaani-project/xz/commit/6e636819e8f0703...

Not at all. I'm talking about running more and more rigorous security tests because you have to catch vulnerabilities, 99% of which are probably introduced accidentally by an otherwise good, reliable developer.

This can be done in multiple ways. A downstream distribution which adds its own layers of security tests and doesn't blindly accept upstream commits. An informal standard on open source projects, kinda like all those Github projects with coverage tests shown on the main repo page. A more formal standard, forcing some critical companies to only adopt projects with a standardized set of security tests and with a sufficiently high score. All these approaches focus on the content, not on the authors, since you can have a totally good-willing developer introducing critical vulnerabilities (not the case here, apparently, but it happens all the time).

On top of that, however, you should also invest in training, awareness, and other "soft" issues that are actually crucial in order to actualy improve cybersecurity. Using the most battle-tested operating systems and kernels is not enough if someone actually puts sensitive data on an open S3 bucket, or if someone only patches their systems once a decade, or if someone uses admin/admin on an Internet-facing website.

Given control of a package which is on most Linux systems and a direct dependency of many things which are not systemd - run apt-cache rdepends liblzma5! – they can choose whatever they want to accomplish that goal. That could be things like a malformed archive which many things directly open or using something similar to this same hooking strategy to compromise a different system component. For example, that includes things like kmod and dpkg so they could target sshd through either of those or, if their attack vector wasn’t critically dependent on SSH, any other process running on the target. Attacking systemd for this is like saying Toyotas get stolen a lot without recognizing that you’re just describing a popularity contest.

1. What "exitcode" is set for:

  exitcode=1
  exit 1

  lklfuse -o type=ext4 "${loop}" "$mnt"
  return $?
  ...
  veracrypt  --text --non-interactive -d "$file"
  return $?
  ...
  mount "$loop" "$mnt"
  return $?

  [ $? = 0 ]
  [ $? != 0 ]
  [ $exitcode = 0 ]

  tomb_init() {
   system="`uname -s`"
   case "$system" in
    FreeBSD)
     cat <<-EOF
     create=posix_create
     format=posix_format
     map=posix_map
     mount=freebsd_mount
     close=freebsd_close
     EOF
     ;;
    Linux)

Maybe their account is compromised, maybe the username borrows the identity of an innocent person with the same name.

Focus on the code, not people. No point forming a mob.

(e: post above was edited and is no longer directed at the person. thanks for the edit.)

They made themselves the primary contact for xz for Google oss-fuzz about one year ago: https://github.com/google/oss-fuzz/commit/6403e93344476972e9...

- Jia Tan <jiat75@gmail.com>

- jiat75 <jiat0218@gmail.com>

``` amap = generate_author_map("xz")

        test_author = amap.get_author_by_name("Jia Cheong Tan")

        self.assertEqual(
            test_author.names, {"Jia Cheong Tan", "Jia Tan", "jiat75"}

        )

        self.assertEqual(

            test_author.mail_addresses,

            {"jiat0218@gmail.com", "jiat75@gmail.com"}

        )

https://github.com/search?q=repo%3Alibarchive%2Flibarchive+j...

It does look innocent enough though. Let's hope there's no unicode trickery involved...

I work on OSS-Fuzz.

As far as I can tell, the author's PRs do not compromise OSS-Fuzz in any way.

OSS-Fuzz doesn't trust user code for this very reason.

> not requiring me to be an expert in another language just to audit it.

Do you do that every time your Cargo.lock changes?

So much for the "facts".

As for trolling: just look at the usual contributions from your community like https://twitter.com/DevuanOrg/status/1619013961629995008 Excellent work with the ad-hominem attacks there.

After all, if it hadn't had a performance regression (someone could submit a PR fixing whatever slowed it down, heh) it still wouldn't be known.

It does remove the safe prefixes... But it also adds one print statement to "strerror()", which could plausibly give better explanations for the error code...

The only suspicious thing here is the lack for safe_ prefix (and the potential for the strerror() function to already be backdoored elsewhere in another commit)

How would an automated CI or build infrastructure stop this attack? It was stopped because the competent package maintainer noticed a performance regression.

In this case, this imagined build system would have to track every rust library used in every package to know which packages to perform an emergency release for.

For example, their email is only registered to GitHub and Twitter. They haven't even logged into their Google account for almost a year. There's also no history of it being in any data breaches (because they never use it).

Burn the witch.

You coulda just pointed out that just because they did right in the case of DSA, doesn't mean we should actually ever trust them, which I would agree is the correct stance.

Mostly I think that story is neat and wanted people to know about it, so I asked a question as a performative writing technique.

Not to mention if you have a Rust application that depends on C libraries, it already dynamically links on most platforms. You only need to rebuild if a Rust crate needs to be updated.

https://news.ycombinator.com/item?id=39874621

> He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.

Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].

[0] - https://github.com/c-skills/rust1

[1] - https://checkmarx.com/blog/over-170k-users-affected-by-attac...

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates".

All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever.

Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.

Does that mean this affects RHEL and Fedora?

My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )

Getting punched in the face is actually a necessary human condition for a healthy civilization.

The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.

Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03

Blog post: "Kuso拍賣．有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>

Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>

The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.

The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.

If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).

Example question 38:

    Question 38
    accounta066 (3): Are these chopsticks really that good? I kind of want to buy
    them! But I recently sent money for online shopping but didn’t receive anything.
    It’s very risky; currently jiat0218 you don’t have any reviews, you can
    interview me. Do you want to hand it over?! … A sincere buyer will keep it.
    
    Reply to
    jiat0218 (4): First of all, I would like to express my condolences to you for
    your unfortunate experience! What can I say about this kind of thing...My little
    sister has always been trustworthy. What’s more, this is a pair of spiritual 
    chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
    my little sister is very careful and takes her time when answering your
    questions. Except for the two messages that were accidentally deleted by her,
    she always answers your questions. If this still doesn’t reassure you, then I
    can only say that I still have room to work hard. You are still welcome
    to bid... ^_^

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...

Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath.

An attacker took over of the project slowly and stayed dormant until recently.

Are they somehow in the clear unless we can show they actively exploited it?

  {0}[calvinow@mozart ~] dpkg-query -W liblzma5
  liblzma5:amd64  5.6.0-0.2
  {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
  1

It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).

I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.

[0] https://pack.ac/

[1] https://news.ycombinator.com/item?id=39793805

[2] https://github.com/PackOrganization/Pack/issues/10

I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

If I am keeping stuff up to date, is there anything at all to worry about?

FYI this looks for pkgs with liblzma:

> dpkg -l |grep liblzma

Versions >= 5.6 are compromised

A supply chain attack serve as the basis for another supply chain attack.

Practically speaking it can't - For one the script injected into the build process tests that you're running on x86-64 linux, for another, the injected code is elf code, which wouldn't link on a mac. It also needs to manipulate dynamic linker datastructures, which would also not work the same on a mac.

> This could perhaps affect more than just sshd on x86-64 linux?

This however is true - /usr/sbin/sshd was the only argv[0] value that I found to "work", but it's possible there are others. "/usr/sbin/sshd" isn't a string directly visible in the injected code, so it's hard to tell.

- linux

- x86-64

- building with gcc & the GNU linker

- part of a .deb or .rpm build

Add to that, as the article explains: openssh does not directly use liblzma, the only reason SSH is affected at all, is because some Linux Distros patch openssh to link it against systemd, which does depend on liblzma.

Could it affect things other than SSH on a Mac? Unlikely. The compromise was introduced in 5.6.0, but macOS Sonoma has 5.4.4 (from August last year).

Nothing except, in no particular order: 1) only having one version of crates 2) mismatched features 3) new transitive dependencies that can be introduced at any time without any warning 4) only supporting one version of rust 5) packages being noarch and basically glorified distro-wide vendoring—so their build.rs code is still run on your machine at cargo build time

That's a common defense for any bloatware. If they're modular and easily disabled then why are they all enabled by default?

Socket activation and the NFS automounter appear to.

If I run "netstat -ap" I see pid 1 listening on enabled units.

Edit: tinysshd is specifically launched this way.

Edit2: there is also substantial criticism of xz on technical grounds.

https://www.nongnu.org/lzip/xz_inadequate.html

A few things:

1. It'd be Cargo.lock

2. Debian, in particular, processes Cargo's output here and makes individual debs. So they've taken advantage of this to already know via their regular package manager tooling.

3. You wouldn't dive into and look through these by hand, you'd have it as a first-class concept. "Which packages use this package" should be table stakes for a package manager.

> Then what? Do I fork each one, bump the version, hope it doesn't break everything, and then push an emergency release!??!

The exact same thing you do in this current situation? It depends on what the issue is. Cargo isn't magic.

The point is just that "which libraries does the binary depend on" isn't a problem with actual tooling.

People already run tools like cargo-vet in CI to catch versions of packages that may have issues they care about.

    commit af071ef7702debef4f1d324616a0137a5001c14c (HEAD -> master, origin/master, origin/HEAD)
    Author: Jia Tan <jiat0218@gmail.com>
    Date:   Tue Mar 26 01:50:02 2024 +0800

        Docs: Simplify SECURITY.md.

    diff --git a/.github/SECURITY.md b/.github/SECURITY.md
    index e9b3458a..9ddfe8e9 100644
    --- a/.github/SECURITY.md
    +++ b/.github/SECURITY.md
    @@ -16,13 +16,7 @@ the chance that the exploit will be used before a patch is released.
     You may submit a report by emailing us at
     [xz@tukaani.org](mailto:xz@tukaani.org), or through
     [Security Advisories](https://github.com/tukaani-project/xz/security/advisories/new).
    -While both options are available, we prefer email. In any case, please
    -provide a clear description of the vulnerability including:
    -
    -- Affected versions of XZ Utils
    -- Estimated severity (low, moderate, high, critical)
    -- Steps to recreate the vulnerability
    -- All relevant files (core dumps, build logs, input files, etc.)
    +While both options are available, we prefer email.

     This project is maintained by a team of volunteers on a reasonable-effort
     basis. As such, please give us 90 days to work on a fix before