I still find this impressive, and it is nice that we now have a test (NixOSTests BTW are awesome, I agree with OP) to avoid this regression from coming back. But from the title it seems to be a widespread issue, not something that affects only one Distro.
Yes, this does not affect people on stock configurations for the plain reason that they wouldn't expect the volume key to be safe during suspend anyway.
Debian's solution was ported to several (most?) other distributions and I guess quite a few people maintained private ports.
The thread-keyring(7) manpage promises: "A thread keyring is destroyed when the thread that refers to it terminates." For their key upload (from userspace to kernelspace) mechanism, the cryptsetup project relied on this property; but kernel 6.9 introduced a regression invalidating this property.
However, if you hibernate (suspend to disk) the entire contents of RAM (including the master key) is written/encrypted to disk and the RAM is cleared.
When you wake the machine up you have to re-enter the passphrase to decrypt the master key to re-load disk contents back to memory.
Up to kernel 6.8, this worked as described; starting with kernel 6.9, it silently didn't.
It was also fun to write, and enabled git-bisecting to isolate the specific kernel refactoring which introduced this bug: https://github.com/NixOS/nixpkgs/pull/532499
The only reason that I do the disk encryption is so that I don't have to worry about people going through my laptop to steal tax documents and/or credit card stuff when I sell the laptop. I of course also wipe the laptop too, but I figure that if the data is encrypted at the drive level then there's very little risk of anyone being able to use some kind of forensics tool and recover data.
The point being made is: If one isn't re-entering their passphrase after suspend, how are they surprised that the encryption keys are somewhere in memory during suspend?
edit: I see now that the prompt was being given and the keys still resided in memory.
Plus what Debian extension to Linux tooling does although nice in theory, but in practice if one really worries about cold-boot attacks, then all keys and important documents has to be wiped out from memory, not only LUKS keys.
So hibernating is really the only proper way to protect against cold boot.
I agree; or resurrecting FridgeLock: https://www.sec.in.tum.de/i20/publications/fridgelock-preven...
AFAIK it's practical only if you make use of TPM. And if you do, you're basically at mercy of TPM.
> where does it get a key to decrypt memory on resume?
- if your CPU supports it, enable memory encryption.
- if your TPM module supports this look for MemoryOverwriteRequestControl & MemoryOverwriteRequestControlLock (/sys/firmware/efi/efivars/) and toggle them. make sure that your computer always reboots and never powers off. memory will always be wiped on boot.
I don't get it. Obviously, the laptop is locked when it resumes, how is that key "for the taking by anyone"? I'm not saying it is impossible to read out RAM from a locked laptop, but surely not by "anyone".
There is a common misconception about how lock-screens in general work - they usually just prevents using the current hardware and software as it is to access the current OS. But the disk encryption is the main thing that prevents modification and other kind of access to actual data. And if the disk encryption key is lying in the memory, then effectively, the disk encryption is bypassed if someone can access the machine physically and assuming that there are no sufficient tampering protections in place for that machine.
Sorry, I'm probably dense, I still don't get it. You steal a laptop, you open it, the screen is locked with a password/fingerprint whatever. How do you read out the RAM from that laptop?
TempleOS is the only thing that comes to mind that doesn't fit your description and it's not practically useful.
Any sufficiently large codebase is a mix of ideas and concepts implemented by different people with different priorities over a large timespan and if you can fit the entire thing in your head it's not very interesting or complex.
Something like disk encryption would be immediately visible.
So you don't have this mess of 80 different distros with 60 different versions of systemd, 20 that don't use it, a million kernel versions and it's all thrown together in a Costco-sized trash bag and we call the output "Linux".
What's the alternative? Proprietary closed-source operating systems owned by corps who can be compelled to insert covert backdoors?
If BSD was as popular as Linux it would have the exact same problems.
It's basically something like this:
original: DoTheThing()
new: DoTheThingSlightlyDifferentButKeepMyCredentialsAlive()
fix: DoTheThingSlightlyDifferentButDoInFactNOTKeepMyCredentialsAlive()
In my experience a substantial portion of gnarly bugs come down to a violation of a high-level system invariant and those do not strike me as something that can be automated. Even with something like Lean you can prove your program satisfies certain properties but you need to have thought about those properties in the first place. The proof doesn't discover the invariant for you.
If you'd had thought about the relevant security property you could have written a regression test for it which is not hard. IMO the really hard part isn't expressing the implementation safely, but it's the realization that this was a property the implementation needed to preserve.
In another thread comment below i argue that maybe the system (OS) itself is so complex that it lacks clear contract / the contract evolves too quickly over time (as other parts of the code need to change the given piece of code to extend it to their use case) and that defies clear encoding?
Or we lack easy enough means to describe specs? I tried reading jepsen spec earlier today and despite it being an "integration test" of sorts, it is far from "simple".
Can an entire OS or a system of comparable complexity be decomposed into objects simple enough that their entire intended behavior (with all edge cases) can be explained in a paragraph of human text + half a screen of dense behavioral "spec" - if i do X and do Y, Z should come out / hold _no matter what happens in-between_. Or that's what asserts + fuzzing is effectively supposed to do? Is there a clear distinction between invalid input and failed invariant in typical C code? I guess error code vs seg fault?
This is a bug involving intersecting concerns and a deficit of cross-domain knowledge. It probably would have been the same in Lisp or assembly language.
The problem is being more auditable does not automatically make it more audited.
There have to be enough people with skill taking enough time to work on it.
The test actually proves it and while it too can change it has more staying power because it's expressed at a higher level of abstraction ("random arcane weird C shit" in the case of code versus "does this property hold" in the case of a regression test).
Because code review is sometimes not much different from an idealized version of the halting problem, where you would have access to a formalized version of a specification.
In other words, there is no strict definition of what is a security issue.
Not a DB person either, but things like TLA+ seem very hard to write even with LLMs. Behavioral tests with an enumerable number of random paths to take (aka model checking - eg jepsen) seem more feasible. Although you can't check internal properties of the system (string `pass` or any of it's copies or parts are not held anywhere in memory at any point between lines A and B) unless we can check that two memory dumps are indistinguishable with different pass strings (assuming we abstracted away storage devices in a test environment).. Also not sure if it's "easy enough" to write such tests either.
Maybe the reason is that OS domain objects / primitives are too complex and not "isolatable" enough / lack a clear contract at all? (Hence multi file refactorings that break invariants.)
Closed source…..
Does such a bug exist in Windows? OSX? Who checks? If someone finds the key in memory, can they tell what conditions might be causing it and where?
Their only recourse under those situations is to hand it off to the OS Vendor and trust that what they implement does solve the problem, and trust that it wasn't a deliberate back-door that is now being replaced by another back-door.
Thanks, that's what I thought.
That's the problem though, right? If it's pointed out we all agree the "do not keep credentials alive" is a property that should hold and we can leverage whatever the environment offers to help preserve it. I fully agree modern languages have amazing support for this, but in C you can still run tests. Let's just say I don't think the language's inability to express logic of this kind held all those involved back from testing for it. I personally find "we just didn't think of it" much more likely.
That said, I am not a fan of C and recommend leveraging whatever fantastic modern tooling is available to you.
https://community.spiceworks.com/t/hard-coded-password-backd...
This sort of thing leads to every kind of exploit, like
https://www.linkedin.com/pulse/half-worlds-fortinet-firewall...
https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...
Other options: DMA attacks. Also you never know what the Intel Management Engine hidden in your computer is doing. It's running a version of Minix you don't have any control over, and it has full access to memory.
the term to look up is "cold boot attack" (https://en.wikipedia.org/wiki/Cold_boot_attack).
tons of cool live demonstrations of how it works on youtube if you've got the 20-40 minutes to spare
If that was the case for the people using the debian extra secure extension that should have wiped the memory clean then someone would have found this bug much earlier than two years. Their password was required to be re-entered even though the key was still in memory somewhere.
I have never had that setup so I was confused.
(You don't mean BitLocker, right?)
But I would never trust it a second, being proprietary and known for issues. You likely know that, but for the benefit of others:
38C3 - Windows BitLocker: Screwed without a Screwdriver https://media.ccc.de/v/38c3-windows-bitlocker-screwed-withou... https://www.youtube.com/watch?v=5eNtT2p12cM
https://www.forbes.com/sites/thomasbrewster/2026/01/22/micro...
BitLocker with a password (the equivalent of the LUKS configuration in question) does not share these issues.
'Happily' is also a stretch, as they really don't have a choice if served a valid court order.
If you want encryption that is safe from the US government, keys need to be stored in your head. Anything physical is subject to court orders.
It's still brittle, awkward and puzzlingly awful UX despite being the literal standard for the platform.
Compare it to any of the actively maintained alternatives, Filevault for MacOS (which is wonderful and never sends your key to be kept somewhere else) or LUKS on Linux.. heck, even Veracrypt is actually easier to understand and more robust.
no, i mean great.
managing a fleet of 100+ laptops with bitlocker is a breeze. its so seemless that the users don't even realize its enabled (i.e. no UX issues, at all).
on the other hand, i am not managing 100+ laptops that use veracrypt. sounds absolutely awful. i've never managed an apple fleet, so i can't speak to that, and will take your word on it.
for personal use, i do not recommend bitlocker (or windows, really), but for already-windows enterprises? absolutely
Did you read the documentation?
https://support.apple.com/guide/mac-help/protect-data-on-you...
"iCloud account: Click “Allow my iCloud account to unlock my disk” if you already use iCloud. Click “Set up my iCloud account to reset my password” if you don’t already use iCloud."
https://developer.apple.com/documentation/devicemanagement/f...
"FileVault Full Disk Encryption (FDE) recovery keys are, by default, sent to Apple if the user requests them. Only one payload of this type is allowed per system."
What is brittle or awkward?
Where is it?
A) Uploaded to microsoft
B) Somewhere in EntraID?
C) Somewhere in our onprem AD?
D) Written down on a scrap of paper when I set up the laptop
the fact that they never ask for the passphrase is a weakness of the system. Because now you have an extremely difficult situation as soon as you're off the happy path.
It's also like 64 characters alphanumeric with no capability to copy/paste.
Compare it to Vera/Filevault where the access key is the users passphrase. In MacOS it's literally your account password, which follows along with your in-OS account credentials.
If you click "Allow my iCloud account to unlock my disk", your recovery key is escrowed to Apple, tied to your Apple Account.
If you don't select that option it never does.
I should have said "without your explicit permission", but I assumed we were all adults and understood that.
The main point is that it's using your account password to unlock, the recovery key is for if you forget your account password.
Brittle is what happens when you haven't logged on to the machine in 60 days, trust with AD is broken, TPM has a glitch and wipes the in device key and forces you into recovery... or god forbid you service the laptop and now you have to enter recovery mode.
Then you're in a nightmare, trying to give someone a super long passphrase over the phone is a not-too-uncommon occurance.
That's assuming you have a good policy for storing the recovery keys. Too loose and they're handed out to everyone, sort of defeating the purpose: too strict and you need the IT department (or specific members), and its still predicated on the notion that you have a policy for it... Given that Admins are a dying breed... I don't think this is workable.
If you compare with Filevault on MacOS: which tracks the credentials of the logged in user; there's no "issue" if the device loses trust because ultimately you always use the real unlock key: not something cached in a "secure storage".
I think both approaches are valid trade-offs and I think that the default Secure Boot BitLocker configuration, for all its architectural tradeoffs, can probably be credited for an enormous amount of data loss mitigation originating from used hard drives alone.
If I as an admin give you your key: it is “leaked” effectively.
hoping users don’t forget their password is a very weak policy.
specifically, the policy and admin points you brought up above, how does veracrypt solve them?