NPM's release cooldown is security theater(blog.outv.im) |
NPM's release cooldown is security theater(blog.outv.im) |
Except there are researchers chomping at the bit to download every new release of packages in search of vulnerabilities they can exchange for fiat, reputational credit, or both.
Anyhow, If you believe LLMs will kill security research, then surely you must also believe a cooldown of 1 week would be enough time for your favorite LLM to analyze a new package version?
I'm confused by this take. In general, LLMs don't replace the expertise required to use an LLM.
Knowing the right input to give the LLM is not enough, nor does it ever end. The LLMs themselves need to be continuously updated and/or specialized for use case, and the "right input" is a moving target.
Keeping both ends of this aligned is the exact same expertise we already had before LLMs. These implementation details would only impact the credibility of those researchers who don't adapt to using this new tool. Who would stumble over something as trivial as this? LLMs aren't even the most important tool in the toolbox anyway.
Different groups have different risks tolerances, and it's absurd to think that nobody is going to pull releases before the default cool down period.
Why is that absurd? It seems like a more likely outcome than just assuming there are enough "security researchers" out there doing this for every package?! IMO both sides here seem to have no evidence that the other is wrong.
You should probably also be paying them directly.
(I agree that companies should pay for security products, of course. The bigger problem there is that so many security products are terrible, and supply chain products appear to be no different as a class.)
Most of the attacks we have seen lately are from legitimate projects which got compromised. The maintainers of those projects are generally looking out for that.
https://www.merriam-webster.com/dictionary/champing%20at%20t...
https://www.merriam-webster.com/dictionary/champ
The migration of "champing at the bit" to "chomping at the bit" is inevitable.
There don't seem to be many idioms like this, where the newer version is both semantically intuitive/meaningful and accepted by authorities. The closest is probably "hone in on", but the verb form of "home" as in "homing device" is still current, unlike the verb form of "champ". Another interesting one is the migration from "just deserts", using an archaic meaning of "desert" as in something deserved, to the modern form "just desserts". However there's much greater distance between "desserts" and the archaic meaning of "deserts" than between "chomp" and "champ".
The most likely explanation is obviously that it's just a bunch of attention seeking amateur. But damn.
Beyond that, a lot of these cooldown criticisms assume everyone will use the same length, which isn't how it has to work. There are people who run Arch because they want the latest version of everything, and there are people who run Debian because they never want a new version to break their system. Some people camp out for the newest iPhone on day 1, and some people wait to see reviews and find out if the antenna is sensitive to how the phone is held. Adjustable cooldowns let everyone choose their own balance between getting the latest features and getting something that is tested.
This ignores all the security companies & researchers screening packages.
Most maintainer are going to notice that themselves.
It will make credential attacks much harder to pull off successfully and the returns from using them will drop pretty dramatically.
One should sandbox all npm command executions.
su -l npm
1. Security researchers and companies are incentivized to install packages early.
2. Even without security patches, people and teams can still set the cooldown period that is appropriate to their risk tolerance. Even if someone in the wild does "get bit", it's better that a smaller number of people get bit than everyone getting bit at the same time. That's not "security theater", that's limiting the blast radius.
I'd also add that adding in cooldown periods lowers the incentives for bad guys to try to compromise a package in the first place because there is a much greater chance all their work will be for naught.
So I started reading this post with interest, thinking it would have some reason I hadn't thought of, but it just boils down to the faulty premise that "Godot is not coming". Bullshit, at the very least I now have agency over where I am in the download timeline. I just don't understand how such bad posts built on the flimsiest of assumptions make it to the top of HN.
And, the complainers are free to set the wait timer to 0 and go back to the way it was. I'll be glad to let them be the first penguin to jump in the water.
Are security researchers going to be doing free labor for you indefinitely for exposure?
If you're going to "DYOR" or whatever, having your automated dependency management on a cooldown timer seems like a snazzy way to smoothly provide a window for all your fancy dependency validation to happen.
I don’t agree with either claim: I don’t think they’re security theater, and I do think they universalize, contingent on there being security parties that are incentivized to scan packages.
The latter is a big assumption, but I think it’s a win for us either way: if supply chain companies are actually capable of defending us, then we win. But if it turns out they can’t serve as the cooldown vanguard, then we have great evidence that they shouldn’t be trusted at all.
> But if it turns out they can’t serve as the cooldown vanguard, then we have great evidence that they shouldn’t be trusted at all.
If they cannot serve as the cooldown vanguard, we are indeed going to realize they are not trustable, but by the time the damage has already been done. Therefore, if companies want to prevent the damage as much as possible, I believe they should do the scan by themselves.
Shall have thought about that... but I'm playing too many Unity games recently.
I actually proposed a cooldown for apt-get in my PhD thesis in 2013 and calculated its impact. Because the meta-data for the Ubuntu apt repository can be completely downloaded you can simulate the impact of a cooldown on how many less packages would be installed and it isn't zero.
There are always people in a community running beta version, or up to date versions. There will be less with a cool down, but then depending on how critical your system is you could adjust cooldown length
Account takeovers, deployment exploits whatever the root cause, given a few days it's incredibly likely maintainers will notice they've released an unexpected update and fix/sound an alarm. A lot of existing cases like this people realise pretty fast, but the packages might still be live for a few hours as NPM catches up. Cooldown entirely removes that 'updated at the wrong time' issue, an no-one even needs to actually look at the code at all.
For inactive or intentionally malicious maintainers, yes it relies on vetting. But it's hardly a mythical 'canary' - there are already a lot of companies scanning every popular npm package, auditing new ones, or just generally analysing threats. I can think of at least 4 such companies without even trying, I'm sure there's loads more.
And they'll continue because it's not warm hearted so much as promoting/testing their security scanning services and tools so they can get the kudos from being the first to spot X etc. Which often does hit the front of HN or reddit, but they don't need the massive cyber loudspeaker to be effective. Flagging the issue to maintainers, reporting to NPM security - with a cooldown in place the package can be voided before anyone really gets the chance to install it. This is literally how a lot of things are spotted already, just with that 'updated at the wrong time' issue catching unlucky people out.
Does it solve everything? No. Will it reduce the amount of problems that make it to end users? Obviously yes.
And for all the author's 'no one will do this', 'waiting for Godot' cries - what do they recommend? Running your own audits, using LLMs, static analysis and testing it yourself. Basically being the veter they claim no-one will be! Unless they don't intend to sound the alarm themselves, they are literally the canary they're looking for.
Cooldowns aren't magic, but they're not security theater either.
There are always early adopters and researchers looking at new, publicly available software. Critical industries have long waited for software releases to be stable and "recommended" by the vendor before upgrading, letting those early adopters who can afford some availability risk take time with it.
The observation that "if everyone in production has a cooldown, the benefit is gone" is an absolutist opinion.
I have seen this occur in some fashion - my company's product has a software version that's been out for almost six months and we still tell customers in critical industries to stay on the older version out of an abundance of caution and with the understanding of what upgrades require for them. Stability > features.
There are COBOL users and CentOS 6 users. They aren't affected by this issue.
They might need to face another set of issues like vulnerability backporting and so on. People not worrying about cooldowns might still need to patch their log4j libraries, but that's another thing. This post is not for them.
> There are always early adopters and researchers looking at new, publicly available software.
I agree with that. These hacks are very unlikely to go unnoticed, especially for popular packages (although they do happen; `chalk` has 3M downloads per week when it was hacked).
> The observation that "if everyone in production has a cooldown, the benefit is gone" is an absolutist opinion.
Yes. However, I think this is eventually becoming some type of pointless arms race, as cooldown does not act actively towards reducing ecosystem attacks.
To add a somehow random example, pnpm's default cooldown time is 1 day. Yarn's 7 days. I'm not sure whether one day pnpm would also change it to 7 days because of people asking.
The nature of some of these attacks, it does. If your developers machine with publishing rights gets rooted a cool down makes credential gathering attacks via script in npm so very much harder it's not even funny.
In some of the recent supply chain attacks the entire fiasco was only a few hours long, but in that time tens of hundreds of thousands of credentials may have been stolen.
One day timers have the weakness of weekends and holidays reducing the number of eyes/systems seeing the issue.
In no circumstances will everyone. This is just a bad take based on a silly premise.
Most, if not all, of the shai-hulud attacks that hit npm and other ecosystems were preventable with cooldowns. And these were not detected because regular users reported the worms, but because security researchers did. I don’t think I’ve ever seen an attack that was discovered because a user reported it.
https://www.npr.org/sections/memmos/2016/06/09/605796769/che...
- Independent security companies are scanning the packages (be careful if the project is depending on some no-so-popular packages)
- Maintainers react promptly (it would be nonsense if people need to cooldown for 21 days because people may have 21-day vacations)
That makes me think LLM impacts human researchers' rewards, but now I realized the result might actually go in the opposite direction according to Jevons paradox. People still need experienced and professional human security researchers after all.
I'm glad you brought up Jevons Paradox, because it's for this reason I believe it's an extremely poor description of what's happening with AI.
The broader topic people should be discussing is not Jevons Paradox, but marginal utility.
More artifacts have diminishing utility.
I’d never heard of socket until they found and reported shai hulud hiding in pytorch lightning. It pays off.
I think there are diminishing returns to be had.
This entire article seems to say that if it doesn't prevent 100% of all possible attacks that it's "security theater". That's nonsense. It's just one more tool in the tool belt, and a highly useful one at that.
> Run LLM-assisted audit on vendored code.
This post is based on an assumption from what I see (I would be very happy if it's wrong) that most companies do not event bother to do these scans. They are merely waiting for the free kindness.