“They introduce kernel bugs on purpose”(lore.kernel.org) |
“They introduce kernel bugs on purpose”(lore.kernel.org) |
Seemed to have posted some clarifications around this. worth a read
I guess it's not as feasible as they thought.
I think there should be a real world experiment to test it.
https://integrity.umn.edu/ethics
Feel free to write to them
New white paper due soon
I don't think this was the pitch they gave to their IRB.
The difference between OSS and closed source is not the number of reviewers for the initial commit, it's the number of reviewers over years of usage.
Greg’s response is totally right.
posted some clarifications around this, worth a read
Here's how the professor (a sociologist) described his methodology:
These three sets of behaviors – rigidly competitive pvp tactics (e. g., droning), steadfastly uncooperative social play outside the game context (e. g., refusing to cooperate with zone farmers), and steadfastly uncooperative social play within the game context (e. g., playing solo and refusing team invitations) – marked Twixt’s play from the play of all others within RV.
Translation: He killed other players in situations that were allowed by the game's creators but frowned upon by the majority of real-life participants. For instance, "villains" and "heroes" aren't supposed to fraternize, but they do anyway. When "Twixt" happened upon these and other situations -- such as players building points by taking on easy missions against computer-generated enemies -- he would ruin them, often by "teleporting" players into unwinnable killzones. The other players would either die or have their social relations disrupted. Further, "Twixt" would rub it in by posting messages like:
Yay, heroes. Go good team. Vills lose again.
The reaction to the experiment and to the paper was what you would expect. The author later said it wasn't an experiment in the academic sense, claiming:
... this study is not really an experiment. I label it as a “breaching experiment” in reference to analogous methods of Garfinkel, but, in fact, neither his nor my methods are experimental in any truly scientific sense. This should be obvious in that experimental methods require some sort of control group and there was none in this case. Likewise, experimental methods are characterized by the manipulation of a treatment variable and, likewise, there was none in this case.
Links:
http://www.nola.com/news/index.ssf/2009/07/loyola_university...
https://www.ilamont.com/2009/07/academic-gets-rise-from-brea...
1) send ethics complaint to the University of Minnesota, and
2) report this to FBI cyber crime division.
I'm repeating myself, but I'm pretty certain the NSA or other intel agencies (Israel, especially, considering their netsec expertise) have already done it in one way or another.
Do you remember the semicolon that caused a big wifi vuln? Hard to really know if it was just a mistake.
I'm going full paranoiac here, but anyway.
You can also imagine the NSA submitting patches to the windows source code, without the knowledge of microsoft, and so many other similar scenarios (android, apple, etc)
Imagine what happens 25 years from now as some ground-breaking security research is being done at Minnesota, and they all groan: "Right, shoot, back in 2021 some dumb prof got us banned forever from submitting patches".
Is there a mechanism for University of Minnesota to appeal, someday? Even murders have parole hearings, eventually.
Incredible that the university researches decided this was a good idea. Has noone in the university voiced concern that perhaps this is a bad idea?
Aaaaand into the kill file they go.
Been a while since I last saw a proper plonk.
People would reach a point where further conversation makes no sense.
So, one would make a kill file entry, and plonk basically communicated that smack the carriage return, enter key with gratifying authority to the user who had earned their place in the kill file, not to be heard from again.
The conversation is over, sort of like a block works today.
Edit: See in the definition I linked where plonk is the sound of some poor soul hitting the bottom of a kill file? I think that is debatable, depending on perspective. The peeps who mentored me onto the net at the beginning explained it as that gratifying press of the CR/LF [ENTER] key.
The sentiment is the same though.
---
plonk /excl.,vt./
[Usenet: possibly influenced by British slang `plonk' for cheap booze, or `plonker' for someone behaving stupidly (latter is lit. equivalent to Yiddish `schmuck')] The sound a newbie makes as he falls to the bottom of a kill file. While it originated in the newsgroup talk.bizarre, this term (usually written "plonk") is now (1994) widespread on Usenet as a form of public ridicule.
----
This particular plonk is proper, not just as an insult, which is the general use case, because the person who earned the "plonking" did so in spectacularly stupid fashion, in the opinion of the "plonker."
Total classic!
On some older TTY's, the two asterisks denoted bold text too, here HN uses it for italics.
Plain text would show the asterisks as the linked exchange showed to us.
Whether some unknown contributor can submit a bad patch isn't so interesting for this type of project. Knowing the payouts for exploits, the question is: how much money would one bad reviewer want to let one past?
So I hear tinfoil is on sale, mayhaps I should stock up.
I mean, the Kernel is now starting to run in cars and even on Mars, and getting those bugs into stable is definitely no achievement one should be proud of.
Sure we infected you with Syphilis without asking for permission first, but we did it for science!
This is obviously the complete opposite of how you should be communicating with someone in most situations let alone when you want something from them.
I have sure been there though so if anything, take this as a book recommendation for 'How to Win Friends & Influence People'.
(this is not really noop, it would add some slight delay)
> I've never read it, but
If you've never read it, maybe just leave it at that.
> manipulating people
You mean "influencing people", like it says right in the title?
It's a book that has helped millions, which is why it continues to be widely recommended.
It's not for everyone. The advice seems obvious to some, which of course is why it can be so valuable for others.
Just have a go at it, it costs less than a euro as an e-book and it read so easily that you’ll be done in no time.
I'm shocked the researchers thought this wasn't textbook a violation of research ethics - we talk about the effects of the Tuskegee Study on the perception of the greater scientific community today.
This is a smaller transgression that hasn't resulted in deaths, but when it's not difficult to have researched ethically AND we now spend the time to educate on the importance of ethics, it's perhaps more frustrating.
The Linux kernel is a very large space with many maintainers. It would be possible to reach out to the leadership of the project to ask for approval without notifying maintainers and have the leadership announce "Hey, we're going to start allowing experiments on the contribution process, please let us know if you'd like to opt out", or at least work towards creating such a process to allow experiments on maintainers/commit approval process while also under the overall expectation that experiments may happen but that *they will be reverted before they reach stable trees*.
The way they did their work could impact more than just the maintainers and affect the reputation of the Linux project, and to me it's very hard to see how it couldn't have been done in a way that meets standards for ethical research.
The wasting time argument is nonsense too its not like they did this thousands of times and beside that, reviewing a intentional bad code is not wasting time is just as productive as reviewing "good" code and together with the patch-patch it should be even more valuable work. It not only or adds a patch it also make the reviewer better.
Yeah it aint fun if people trick you or point out you did not succeed in what you tried to do. But instead of playing the victim an play the unethical human experiment card maybe focus on improving.
Ridiculous. Does the same apply to pentesting a bank or a government agency. If you wanted to pentest these of course you'd get approval from an executive that has power to sanction this. Why would Linux development be an exception? Just ask GKH or someone to allow you to do this.
Problem with that is it's a lot of work and they didn't want to do it in the first place.
Yes. Someone sees the work provided to the community for free and thinks that gives them some ethical privilege to put that work to the test?
To the supply chain type of attacks, there isn't an easy answer. Classical methods left both the SolarWinds and Codecov attacks in place for way too many days.
yes, this is pentesting
Unless there's something particularly different about University of Minnesota compared to other universities, something tells me that they won't give a crap unless you're a donor.
"Yes, we are banned from submitting patches to Linux due to past academic research and activities of our PhD students. However, we have a world-class program here."
I'm glad I never contributed again as an alumni...
If enough grads do that, I would expect the university will do something about it, and that would send a message. It's about where the money comes from in the end; (tuition, grants, research partnerships etc) IMO none of these sources would be very happy about what might amount to defacement of public property and waste of the time of people that are working for the good of mankind by providing free tools(bicycle of the mind) to future generations.
There is no novelty in this research; bad actors have been trying to introduce bad patches for as long as open source has been open.
He himself is to blame for submitting these kind of patches and claiming innocence. If a person as old as him can't figure out what's ethical and what's not, then that person deserves what comes out of actions like these.
But the opposite of what you propose is true. The maintainers are annoyed by others wasting their time in other cases as well as in this case - it's coherent behavior. And in my opinion, it's sensible to be annoyed when someone wasted your time - be it by lazily made patches or by intentionally broken patches.
I suspect that all of the issues I'm aware of are within normal bounds for any university of that size. That is if kill the UMN you also need to kill Berkley, MIT, and Harvard for their issues of similar magnitude that we just by chance haven't heard about. This is a guess though, I don't know how bad things are.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
One of the reason this worked is likely that submissions from large US research universities get a "presumptive good faith" pass. A small company in the PRC, for an example, might see more intensive review. But given the history of open source, we trust graduate students maybe more than we should.
[1] Originally legal/copyright driven and not a security feature, though it has value in both domains.
Which is a bit silly, isn't it? Grad students are poor and overworked, it seems easy to find one to trick/bribe into signing off your code, if you wanted to do something malicious.
In late January I submitted a patch with no prior contributions, and it was pushed to drm-misc-next within an hour. It's now filtered it's way through drm-next and will likely land in 5.13.
It trashes University of Minnesota in the press. What is going to happen is that the president of the university now is going to hear about it, so will the provost and so will people in charge of doling money. That will rapidly fix the professor problem.
While people may think that tenure professors get to do what they want, they never win in a war with a president and a provost. That professor is toast. And so are his researchers
At least it might prompt the University to take action against the researchers.
The next batch of "researchers" won't be attending the University of Minnesota, and other universities scared of the same fate (missing out on tuition money) will preemptively ban such research themselves.
"Effective" isn't binary, and this is a move in the right direction.
I'm by no means an security expert nor a kernel contributor but considering he's program comitee, is these kind of practices a common place in Security/Privacy researchers?
Does idea/practises like this get a pass on conference publishing regularly?
[0] https://www-users.cs.umn.edu/~kjlu/ [1] https://www.ieee-security.org/TC/SP2022/cfpapers.html
Sure, this is "just" a university research project this time. And sure, this is done in bad taste.
But there are legitimately malicious national actors (well, including the US govt and the various 3 letter agencies) that absolutely do this. And the national actors are likely even far more sophisticated than a couple of PhD students. They have the time, resources and energy to do this over a very long period of time.
I think on the whole, this is very net positive in that it reveals the vulnerability of open source kernel development. Despite, how shitty it feels.
So if only Linus would have listened we would have Linux as microkernel equally feature rich and widespread? Stupid Linus /s
We have an established legal framework to do this. It's called "tort law," and we need to learn how to point it at people who negligently or maliciously create and or mess with software.
What makes it difficult, of course, is that not only should it be pointed at jerk researchers, but anyone who works on software, provably knows the harm their actions can or do cause, and does it anyway. This describes "black hat hackers," but also quite a few "establishment" sources of software production.
https://twitter.com/UMNComputerSci/status/138494868382169497...
Now if we can only find more open source developers to punish for trusting contributors!
Enjoy your ban.
Sorry if this comment seems off base, this research feels like a low blow to people trying to do good for a largely thankless job.
I would say they are violating some ideas of Ken Thompson: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
For example, in economics departments there is usually a ban on lying to experiment participants. Many of them even explicitly explain to participants that this is a difference between economics and psychology experiments. The reason is that studying preferences is very important to economists, and if participants don’t believe that the experiment conditions are reliable, it will screw the research.
Suggested title:
“Linux Kernel developers found to reject nonsense patches from known bad actors”
What happens if any of that patches ends up in a kernel release?
It's like setting random houses on fire just to test the responsiveness of local firefighters.
It had a high human component because it was humans making many decisions in this process. In particular, there was the potential to cause maintainers personal embarrassment or professional censure by letting through a bugged patch.
If the researchers even considered this possibility, I doubt the IRB would have approved this experimental protocol if laid out in those terms.
Imagine how downstream consumers of the kernel could be affected. The kernel is used for some extremely serious applications, in environments where updates are nonexistent. These bad patches could remain permanently in situ for mission-critical applications.
The University of Minnesota should be held liable for any damages or loss of life incurred by their reckless decision making.
However, doing it repeatedly with real names seems not helpful to the community and indicates a questionable motivation.
The benefit is twofold: (a) it's simpler to block a whole university than it is to figure out who the individuals are and (b) this sends a message that there is some responsibility at the institutional level.
The risk is that someone writing from that university address might have something that would be useful to the software.
Getting patches and pull-requests accepted is not a guaranteed. And it's asking a lot of kernel developers that they check not just bad code but also for badly-intended code.
I had a look at the research paper (https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...) and it saddens me to see such a thing coming out of a university. It's like a medical researcher introducing a disease to see whether it spreads quickly.
I fail to see how this does not amount to vandalism of public property. https://www.shouselaw.com/ca/defense/penal-code/594/
The similarities are that reviewers can get sleepy no matter what they're reviewing. Troll doll QC staff get sleepy. Nuclear reactor operators get sleepy too.
Most people in the outgroup who know about the Sokal Affair but who know nothing about the journal they submitted to aren't aware of this, but Social Text was known to be not peer reviewed at the time. It's not that reviewers failed some test; there explicitly and publicly wasn't a review process. Everyone reading Social Text at the time would have known that and interpreted contents accordingly, so Sokal didn't demonstrate anything of value and was just being a jackass.
Getting banned from Linux contribution is an ouchy for the university, but the damage has been done.
Your suggested "wrongdoing by being lazy" is completely made-up nonsense.
https://github.com/torvalds/linux/blob/master/Documentation/...
Right? It's true that all systems can be gamed and you could no doubt fool the right maintainer to take a patch from a fraudulent source. But the point is that it's not as simple as this grad student just resubmitting work under a different name.
Maybe?
My point with the above comment was more to point out that there is no special '"presumptive good faith" pass' that comes along with a .edu e-mail address, not that it's possible to subvert the system (that's already well known).
Everyone, including some random dude with a Hackers (1995) reference for an e-mail address (myself) gets that "presumptive good faith" pass.
If this is foolproof, then no-one should be talking about the replication crisis.
People don't do bad things _expecting_ to be caught, if they haven't already convinced themselves they're not doing anything bad at all. And I suspect it's surprisingly easy to convince people that they won't get caught.
Replication is really a different problem. It's possible for you to do nothing wrong, run hundreds of trials, get a great result and publish it. But it was due to noise/error/unknown factors, and can't be replicated. The crisis is also that replication receives no academic recognition.
When people fabricate results they know it's an offence, the problem with these guys is they don't even acknowledge/understand the ethical rule they are breaking.
it's good work and i'm glad they've done it, but that's depressing.
now what?
Unfortunately even if the latest submissions were sent with good intentions and have nothing to do with the bug research, the University has certainly lost the trust of the kernel maintainers.
> I respectfully ask you to cease and desist from making wild accusations that are bordering on slander.
> These patches were sent as part of a new static analyzer that I wrote and it's sensitivity is obviously not great. I sent patches on the hopes to get feedback. We are not experts in the linux kernel and repeatedly making these statements is disgusting to hear.
> Obviously, it is a wrong step but your preconceived biases are so strong that you make allegations without merit nor give us any benefit of doubt. I will not be sending any more patches due to the attitude that is not only unwelcome but also intimidating to newbies and non experts.
This idiot should be banned from the University, not from the linux kernel.
I back your decision and fuck these people. I will additionally be sending a strongly worded email to this person, their advisor and their whoever's in charge of this joke of a computer science school. Sometimes I wish we had the ABA equivalent for computer science.
Even if you're justifiably steaming about something, please wait to cool down before posting here.
We detached this subthread from https://news.ycombinator.com/item?id=26889743.
The researchers should not have done this, but ultimately it's the faculty that must be held accountable for allowing this to happen in the first place. They are a for-profit institution and should not get away with harassing people who are contributing their personal time. So nail them to the proverbial cross but make sure the message is heard by those who slipped up (not the researchers who should have been stopped before it happened).
A real malicious actor is going to be planted in some reputable institution, creating errors that look like honest mistakes.
How do you test if the process catches such vulnerabilities? You do it the just the way that these researchers did.
Yes, it creates extra homework for some people with certain responsibilities, that doesn't mean it's unethical. Don't shoot the messenger.
They introduced a real vulnerability in a codebase that lowers world-wide cybersecurity used by billions so they could jerk themselves off over a research paper.
They are a real malicious actor and I hope they hit by the CFAA.
We made a mistake. I'm not sure what happened but it's possible that we mistook this post for garden-variety mailing-list drama. A lot of that comes up on HN, and is mostly not interesting; same with Github Issues drama.
In reality, this post is clearly above that bar—it's a genuinely interesting and significant story that the community has a ton of energy to discuss, and is well on topic. I've restored the thread now, and merged in the dupe that was on the front page in its stead.
Sorry everybody! Our only priority is to serve what the community finds (intellectually) interesting, but moderation is guesswork and it's not always easy to tell what's chaff.
Edit: turns out it was just that there were two different threads on the frontpage about this story and a moderator downweighted the earlier one. That's standard moderation. Usually we merge the threads (and I've since done so) but I'm the only mod who currently does that and I wasn't online yet.
It's clear to me now that this case was a moderation mistake. We make them sometimes (alas), but that's also been true for many years. Moderation is guesswork. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Do you understand how dumb that sounds?
We detached this subthread from https://news.ycombinator.com/item?id=26890035.
If you make a dumb analogy, that's on you.
You really think the Linux kernel guys would change their process if you did this? They'd still do the same things they do.
Given the size and complexity of the Linux (/GNU) codeworld, I have to wonder if they are coming up against (or already did) the practical limits of assuring safety and quality using the current model of development.
Someone should look into who sponsored this research. Was there a state agent?
https://experts.umn.edu/en/organisations/confucius-institute
Pentesting unwitting participants is malicious, and in many cases illegal.
A security researcher doesn't just delete a whole hard drive's worth of data to prove they have the rights to delete things, they are trusted for this reason.
They were almost certainly expecting an obvious bad patch to be reverted while trying to sneak by a less obvious one.
Other than that, they got caught red-handed accepting shit patch and complain about ethical issues when the fault is entirely on their side for not doing their job properly.
This whole thing points to a single question: how many times did they accept patch from black hat individuals who did not disclose their intention ?
This question the Linux development security model and highlight it being insecure to such social engineering attacks and they still manage to play victims. That's pitiful... Own it, say you fucked up accepting the patch, don't blame other for your own incompetence.
Maybe I'm just too cynical and paranoid though.
I think that is thinking too kind of them. Sociopaths are often very well-versed to give "reasons" about what they do, but at the core it is powerplay.
The system appears to have worked, so that's good news for Linux. On the other hand, now that the university has been banned, they won't be able to find holes in the process that may remain, that's bad news for Linux.
When a university submits intentionally buggy patches to the Linux Kernel, and the maintainers successfully detect it, the community responds with "That was an incredibly scummy thing to do."
I sense a teachable moment, here.
If this was Facebook and their response was: > ~"stop wasting our time" > ~"we'll report you" the responses here would be very different.
If you look at the website of the PhD student involved [1], they seem to be writing mostly legitimate papers about, for example, using static analysis to find bugs. In this kind of research, having a good reputation in the kernel community is probably pretty valuable because it allows you to develop and apply research to the kernel and get some publications/publicity out of that.
But now, by participating in this separate unethical research about OSS process, they've damaged their professional reputation and probably setback their career somewhat. In this interpretation, their other changes were made in good faith, but now have been tainted by the controversial paper.
> They obviously were _NOT_ created by a static analysis tool that is of any intelligence, as they all are the result of totally different patterns, and all of which are obviously not even fixing anything at all. So what am I supposed to think here, other than that you and your group are continuing to experiment on the kernel community developers by sending such nonsense patches?
Greg didn't think that the static analysis excuse could be legitimate as the quality was garbage.
HN: let's hate researcher(s) instead of process
Wow.
Assume good faith, I guess?
Universities are places with lots of different students, professors, and different people with different ideas, and inevitably people who make bad choices.
Universities don't often act with a single purpose or intent. That's what makes them interesting. Prone to failure and bad ideas, but also new ideas that you can't do at corporate HQ because you've got a CEO breathing down your neck.
At the University of Minnesota there's 50k+ students at the Twin Cities campus alone, 3k plus instructors. Even more at other University of Minnesota campuses.
None of those people did anything wrong. Putting the onus on them to effect change to me seems unfair. The people banned didn't do anything wrong.
Now the kernel doesn't 'need' any of their contributions, but I think this is a bad method / standard to set to penalize / discourage everyone under an umbrella when they've taken no bad actions themselves.
Although I can't put my finger on why, this ban on whole swaths of people in some ways seems very not open source.
The folks who did the thing were wrong to do so, but the vast majority of people now impacted by this ban didn't do the thing.
"We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected."
The current message in the tree is highlighted with the indicator "[this message]"; you can see replies branch out below it and parent messages above it.
It's super unclear.
https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
> On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits
> Qiushi Wu, and Kangjie Lu.
> To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
> Note: The experiment did not introduce any bug or bug-introducing commit into OSS. It demonstrated weaknesses in the patching process in a safe way. No user was affected, and IRB exempt was issued. The experiment actually fixed three real bugs. Please see the clarifications[2].
1: https://www-users.cs.umn.edu/~kjlu/
2: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
This was a bold and unwise exercise, especially if you’re an academic in country on a revocable visa who participated.
By submitting their bad code to the actual Linux mailing list, they have made Linux kernel developers part of their research without their knowledge or consent.
Some of this vandalism has made it down into the Linux kernel already. These researchers have sabotaged other people's software for their personal gain, another paper to boast about.
Had this been done with the developers' consent and with a way to pull out the patches before they actually hit the stable branches, then this could have been a valuable research. It's the way that the research was carried out that's the problem, and that's why everybody is hating on the researches (rather than the research matter itself).
I see it as similar to
- allowing recording of people without their consent (or warrant),
- experimenting on PTSD by inducing PTSD without people consent,
- or medical experimentation without the subject consent.
And the arguments about not having anyone know:
Try to introduce yourself in the White House and when you get caught tell them "I was just testing your security procedures".
One is that what the researchers did is beyond reckless. Some of the bugs they've introduced could be affecting real world critical systems.
The other issue is that the research is actually good in proving by practical means that pretty much anyone can introduce vulnerabilities into software as important and sensitive as the Linux kernel. This hurts the industry confidence that we can have secure systems even more than it already is.
While some praise may be appropriate for the latter, they absolutely deserve the heat they're getting for the former. There may be many better ways to prove a point.
But let's assume your girlfriend points an (unknown to you) empty gun at your head, because she wants to know how you will react. What do you think is the appropriate reaction?
At a cost to mostly people who didn't / and I'll even say wouldn't do the bad thing.
UMN hasn't admitted to any wrongdoing. The professor wasn't punished in any form whatsoever. And they adamantly state that their research review processes are solid and worked in this case.
An indefinite ban is 100% warranted until such a time that UMN can demonstrate that their university sponsored research is trustworthy and doesn't act in bad faith.
I do, because the university needs to dismiss everyone involved, sever their connections with the institution, and then have a person in a senior position email the kernel maintainers with news that such has taken place. At which time the ban can be publicly lifted.
There are ways to do research like this (involve top-level maintainers, prevent patches going further upstream etc.), just sending in buggy code on purpose, then lying about where it came from, is not the way. It very much is wrong in my opinion. And like some other people pointed out, it could quite possibly be a criminal offense in several jurisdictions.
This is what I can't grok. Why would you not contact GKH and work together to put a process in place to do this in an ethical and safe manner? If nothing else, it is just basic courtesy.
There is perhaps some merit to better understanding and avoiding the introduction of security flaws but this was not the way to do it. Boggles the mind that this group felt that this was appropriate behavior. Disappointing.
As far as banning the University, that is precisely the right action. This will force the institution to respond. UMN will have to make changes to address the issue and then the ban can be lifted. It is really the only effective response the maintainers have available to them.
If people affilated with the UMN want to contribute to the Linux kernel, they can still do that on a personal title. They just can't do it as part of UMN research, but given that UMN has demonstrated they don't have safeguards to prevent bad faith research, that seems reasonable.
Note that I suspect enough people have gotten notice by the press now.
Some of the people banned didn't do anything wrong. Others tried to intentionally introduce bugs into the kernel. Their ethics board allowed that or was mislead by them. Obviously they are having serious issues with ethics and processes.
I'm sure the ban can be reversed if they can plausibly claim they've changed. Since this was apparently already their second chance and they've been reported to the university before and the university apparently decided not to act on that complaint ... I have some doubts that "we've totally changed. This time we mean it" will fly.
How many people didn't and did? The numbers seem absurd.
It is not always easy to identify who works for who at a university in regards to someone's research. The faculty member who seems to be directing this is identifiable, obviously. But it is not so easy to identify anyone acting on his behalf - universities don't maintain public lists of grad or undergrad students working for an individual faculty member. Ad in that there seems to be a pattern of obfuscating these patches through different submission accounts specifically to hide the role of the faculty advisor (my interpretation of what I'm reading).
Putting the onus on others is unfair...but from the perspective of Kernel.org, they do not know who from the population is bad actors and who isn't. The goal isn't to penalize the good folks, the goal is to prevent continued bad behavior under someone elses name. Its more akin to flagging email from a certain server as spam. The goal of the policy isn't to get people to effect change, its to stop a pattern of introducing security holes in critical software.
It is perfectly possible that this was IRB approved, but that doesn't necessarily mean the IRB really understood the implications. There are specific processes for research involving deception and getting IRB approval for deception. but there is no guarantee that IRB members have the knowledge or experience with CS or Open Source communities to understand what is happening. The backgrounds of IRB members vary enormously...
It's unfortunate that many people will get caught up in this ban that had nothing to do with it, but the university deserves to take a credibility hit here. The ball is now in their court. They need to either make things right or suffer the ban for all of their staff and students.
I'm pretty sure that if someone from the University of Minnesota would like to contribute something of value to the Linux kernel, dropping a mail to GregKH will result in that being possible.
https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
That looks like quite some significant effort. Now if most of those fixes were real, now after the revert there will be 190 known bugs in the kernel, before it's all cleaned up. That may have some cost too.
Looks like a large and expensive mess someone other than that UNI will have to clean out, because they're not trustworthy, ATM.
Someone wants to introduce bugs, they can.
Meanwhile lots of people banned for some other person's actions.
Even worse: They bragged about it, then sent a new wave of buggy patches to see if the "test subjects" fall for it once again, and then tried to push the blame on the kernel maintainers for being "intimidating to newbies".
This is thinly veiled and potentially dangerous bullying.
Which itself could be the basis of a follow up research paper. The first one was about surreptitiously slipping vulnerabilities into the kernel code.
There's nothing surreptitious about their current behavior. They're now known bad actors attempting to get patches approved. First nonchalantly, and after getting called out and rejected they framed it as an attempt at bullying by the maintainers.
If patches end up getting approved, everything about the situation is ripe for another paper. The initial rejection, attempting to frame it as bullying by the maintainers (which ironically, is thinly veiled bullying itself), impact of public pressure (which currently seems to be in the maintainers' favor, but the public is fickle and could turn on a dime).
Hell, even if the attempt isn't successful you could probably turn it into another paper anyway. Wouldn't be as splashy, but would still be an interesting meta-analysis of techniques bad actors can use to exploit the human nature of the open source process.
Fortunately, the episode also suggests that the kernel-development immune-system is fully-operational.
As soon as I read that all sympathy for this clown was out the window. He knows exactly what he's doing.
I don't have data to back this up, but I've been around a while and I can tell you papers are rejected from conferences for ethics violations. My personal observation is that infosec/cybersecurity academia has been steadily moving to higher ethical standards in research. That doesn't mean that all academics follow this trend, but that unethical research is more likely to get your paper rejected from conferences.
Submitting bugs to an open source project is the sort of stunt hackers would have done in 1990 and then presented at a defcon talk.
IEEE seems to have no problem with this paper though.
>>> On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits Qiushi Wu, and Kangjie Lu. To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
I can think of a whole lot of three letter agencies with reasons to do that, most of whom recruit directly from universities.
Could a number of seemingly unrelated individuals introduce a number of bugs over time to form and exploit without being detected?
"Can open-source maintainers make a mistake by accepting faulty commits?"
In addition to being scummy, this research seems utterly pointless to me. Of course mistakes can happen, we are all humans, even the Linux maintainers.
Both are unethical, disruptive, and prove nothing about the integrity of the organizations they target.
Submitting a fake paper to a journal read by a few dozen academics is a threat to someones ego. It is not in the same ballpark as a threat to IT infrastructure everywhere.
https://duckduckgo.com/?q=facebook+emotional+study&t=fpas&ia...
Any pentester or red team considers their profession an ethical one.
By the response of the Linux Foundation, this is clearly not authorized nor falling into any bug bounty rules/framework they would offer. Social engineering attacks are often out of bounds for bug bounty - and even for authorized engagements need to follow strict rules and procedures.
Wonder if there are even legal steps that could be taken by Linux foundation.
Once they clean out the garbage in the Comp Sci department and their research committee that approved this experiment, we can talk.
However, zooming out a little, I think it's kind of useful to look at this as an example of the incentives at play for a regulatory bureaucracy. Comments bemoaning such bureaucracies are pretty common on HN (myself included!), with specific examples ranging from the huge timescale of public works construction in American cities to the FDA's slow approval of COVID vaccines. A common request is: can't these regulators be a little less conservative?
Well, this story is an example of why said regulators might avoid that -- one mistake here, and there are multiple people in this thread promising to email the UMN IRB and give them a piece of their mind. One mistake! And when one mistake gets punished with public opprobrium, it seems very rational to become conservative and reject anything close to borderline to avoid another mistake. And then we end up with the cautious bureaucracies that we like to complain about.
Now, in a nicer world, maybe those emails complaining to the IRB would be considered valid feedback for the people working there, but unfortunately it seems plausible that it's the kind of job where the only good feedback is no feedback.
This news makes me wish to implement my own block on the same contributors to any open source I'm involved with. At the end of the day, their ethics is their ethics. Those ethics are not Linux specific, it was just the high profile target in this instance. I would totally subscribe to or link to a group sourced file similar to a README.md or CONTRIBUTORS.md (CODERS_NON_GRATA.md?) that pulled such things.
There is also a more nuclear option which I'm specifically not advocating for quite yet here but I will note none the less;
We're starting to see in discourse regarding companies co-opting open source projects for their own profit (cough Amazon) and how license agreements limit them more than regular contributors. That has come about, at the core of it, also because of a demonstrated trend of bad faith but also combined with a larger surface area contact with society. I could foresee a potential future trend where individuals who also act in bad faith are excluded from use of open source projects through their licenses. Imagine if the license for some core infrastructure tech like a networking library or the Linux kernel banned "Joe Blackhat" from using python for professional use. Now he still could, but in reputable companies, particularly larger ones with a legal department that person would be more of a liability than they are worth. There can be potentially huge professional consequences of a type that do not currently exist really in the industry.
At least both of them they are free from such @umn.edu commits with fantasy names.
These patches look like bombs under bridges to me.
Do you believe that some open source projects should have legal protection against such actors? The Linux Kernel is pretty much a piece of infrastructure that keeps the internet going.
In addition to wasting people's time, you are potentially messing with software that runs the world.
Apart from some perhaps critical unsafe stuff which should have a lot of attention, requiring everything to be safe/verified to some extent surely is the answer.
Usually people are admired here for finding vulnerabilities in all sorts of systems and processes. For example, when someone submits a false paper to a peer-reviewed journal, people around here root for them; I don't see complaints about wasting the time and violating the trust of the journal.
But should one of our beloved institutions be tested - now it's an outrage?
This is more comparable to DDOS ing a web server to test their capabilities of handling DDOS. And they are aware of the issue. And they told you to not do it when you did it before. You just don't waste other people's time/money like that unless they give you the permission.
But it means the regularly 'security' research does ethically questionable stuff.
IRBs exist because of legal risk. If parties harmed by unethical computer science research do not litigate (or bring criminal complaints, as applicable) the university practices will not substantially change.
1. You don't conduct a penetration test without permission to do so, or without rules of engagement laying out what kinds of actions and targets are permitted. The researchers did not seek permission or request RoE; they tried to ask forgiveness instead.
2. You disclose the vulnerabilities immediately to the software's developers, and wait a certain period before revealing the vulns to the public. While the researchers did immediately notify the kernel dev team in 3 cases, there's apparently another vulnerable commit that the researchers didn't mention in their paper and did not tell the kernel dev team about, which was still in the kernel as of the paper's publish date.
Apparently the IRB team that reviewed this project decided that no permission was needed because the experiment was on software, not people--even though the whole thing hinged on human code review practices. It's evident that the IRB doesn't know how infosec research should be conducted, how software is developed, or how code review works, but it's also evident that the researchers themselves either didn't know or didn't care about best practices in infosec.
https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
It has yet to be published (due next month)
How about opening few bug reports to correctly report the final response of the community and the actual impact?
Not asking to harass them: if anyone should do it, it would be the kernel devs, and I'm not one of them
The way the university did this tests and the reactions afterwards are just bad.
What I see here and what the Uni of Minnesota seem to neglected is: 1. Financial damage (time is wasted) 2. Ethical reasons of experimenting with human beings
As a result, the University should give a clear statement on both and should donate a generous amount of on money for compensation of (1.)
For part (2.), a simple bit honest apology can do wonders!
---
Having said that, I think there are other and ethically better ways to achieve these measurement.
Researcher sends bogus patches to bazaar-style project, gets them reviewed and approved, uses that to point how ridiculous the review process of the project is => DON'T DO THAT! BAD RESEARCHER, BAD!
Thought to be fair, it is also the case that only the most irrelevant journals are likely to accept the most bogus papers. But in both cases I see no reason not to point it out.
The two situations are much more closer than what you think. The only difference I see is in the level of bogusness.
The vulnerability is in the process, and this was the test.
> You really think the Linux kernel guys would change their process if you did this? They'd still do the same things they do.
If they're vulnerable to accepting patches with exploits because the review process fails, then the process is broken. Linux isn't some toy, it's critical infrastructure.
No, you can't, because that is the test! If you manage to push exploits to the real kernel, the test failed. If you get caught, the test passes. They did get caught.
The way I understand it is that unnecessarily angry or confrontational posts tend to lower the overall tone. They are cathartic/fun to write, fast to write, and tend to get wide overall agreement/votes. So if they are allowed then most of the discussion on a topic gets pushed down beneath that sort of post.
Hence why we are asked to refrain, to permit more room for focused and substantive discussion.
Edit: dang is a good person and I don't understand how he's taking sides here with people sending out malware (because that's what this sums up to). I understand I came on a little hot, but that was unexpected.
It's common when we post a moderation reply for people to assume that the mods are disagreeing with them [1], when we're just asking them to follow the site guidelines. Those two things are orthogonal, but they fuse under temperature: that is, when one is feeling hot about something, it's hard to separate them. They come unstuck as things cool down.
(I don't mean to pick on you personally. This kind of reaction happens in everyone, certainly including me.)
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
After reviewing the thread I don't see any of what you are asking, here, upstream. I don't seem dang coming out on the same side as people sending out malware, and I don't really see that question present. I wish I had something more concrete to say, but I think your take here (and only here) is wrong and that you might have just entered this one on the wrong foot?
"Next" goes approximately down the tree in the order it's displayed on the page, by depth-first search.
"Prev" just reverses the same process as "Next".
"Parent" differs from "prev" in that it goes to the parent e-mail even if this email has earlier siblings.
(Generally, I just scroll down to the tree view and click around manually.)
1) The email message, with a few headers included
2) A thread overview, with all emails in the thread
3) Instructions on how to reply
4) Information about how to access the list archives.
You need only care about (1) and (2). The difference between prev and parent is indicated by the tree view in (2). The previous one is the previous one in the tree, which might not necessarily be the parent if the parent has spawned earlier replies.
No it doesn't, not unless most comments typicaly get upvoted, which seems counterintuitive to me.
A bad comment to downvote ratio indicates a flamewar, since there are no flamewars without downvotes, but more comments than upvotes just means high comment velocity (which can go either way) or just that no one is saying anything particularly interesting, which isn't implicitly harmful.
A flamewar detector that hides popular threads to suppress engagement just in case there might be a flamewar is working at cross purposes with the goal of a forum, which is engagement.
This message contradicted that.
They may not have been included in a release, but should gkh not have intervened *this would have reached users*, especially if the researchers weren't apparently aware their commits were reaching stable.
Since it's an institutional issue (otherwise it would've stopped after they were reported the first time), it doesn't seem wrong to also deal with the institution.
Do you mean that when a small minority commit an abuse (edit: questionable here), the whole group should be condemned ? Me-think HN is as hypocrite as can be on this subject...
This is what you do as a grownup and the other side is expected to honor your request and perform the same thing they do for other commits... the problem is that people think of pen testing as an adversarial relationship where one person needs to win over the other one.
I guess you could receive "authorization" from a confidante who then delegates the work to unwitting reviewers, but then you could make the same "ethical" argument.
Again, from a hacker ethos perspective, none of this was unethical. From a "research ethics committee", maybe it was unethical, but that's not the standard I want applied to the Linux kernel.
What's the benefit? You raise trust in the process behind one of the most critical pieces of software.
I'm not saying people or institutions cant change. But the burden of proof is on them now to show that they did. A good first step would be to acknowledge that there IS a good reason for doubt, and certainly not whine about 'preconceived bias'.
It is not the job of the kernel maintainers to justify the teams new nonsense patches. If the team has stopped being bullshit, they should defend the merit of their own patches. They have failed to do so, and instead tried to deflect with recriminations, and now they are banned.
The Linux kernel has limited resources and if one university lack of oversight is causing the whole process to be stretched tinner than it already is then a ban seems like a valid solution.
> It's not a ban on people, it's a ban on the institution that has demonstrated they can't be trusted to act in good faith. If people affilated with the UMN want to contribute to the Linux kernel, they can still do that on a personal title. They just can't do it as part of UMN research, but given that UMN has demonstrated they don't have safeguards to prevent bad faith research, that seems reasonable.
https://twitter.com/FiloSottile/status/1384883910039986179
(Clearly the academic behavior is also a problem, there's no good justification for asking for reviews of known bad patches)
If this was Facebook and not Linux everyone would look upon this very differently.
There are ways to test social vulnerabilities (pentesting) and they all involve asking for permission first.
I did a bit of walking away, had some water and feel a little better. It's sometimes difficult to separate things from being personal. Sorry, this whole thread has my BP going a little hot because of some things like this I've had to deal with.
It's not you, I probably went a little far there with "fuck these people".
Doing code review at work I am constantly catching blatantly obvious security bugs. Most developers are so happy to get the thing to work, that they don't even consider security. This is in high level languages, with a fairly small team, only internal users, and pretty simple code base. I can't imagine trying to do it for something as high stakes and complicated as the kernel. Not to mention how subtle bugs can be in C. I suspect it is impossible to distinguish incompetence from malice. So aggressively weeding out incompetence, and then forming layers of trust is the only real defense.
If all 10 are rejected but only one had a security concern, then the process is faulty in another way.
Edit: There is this theory that penetration testing is adversarial but in the real world people want the best outcome for all. The kernel maintainers are professionals so I would expect the same level of caring for a "special PR" versus a "normal PR"
I don't know enough about the kernel's process to comment on whether the same approach could be taken there.
Alternatively, if the time window is broad enough, perhaps you could be almost totally open with everyone, withholding only the identity of the submitter. For a sufficiently wide time window, Be on your toes for malicious or buggy commits doesn't change the behaviour of the reviewers, as that's part of their role anyway.
It's far more likely that professor is so out of touch that they honestly think their behavior is acceptable.
What's the process then? I doubt there is such a process for the Linux kernel, otherwise the response would've been "you did not follow the process" instead of "we don't like what you did there".
Firstly, it accomplishes nothing. We already all know that PRs and code submission is a potential vector for buggy code or security vulnerabilities. This is like saying water is wet.
Secondly, it wastes the time of the people working on the linux kernel and ruins the trust of code coming from the university of minnesota.
All of this happened due to caring about one's own research more than the ethics of doing this sort of thing. And continuing to engage in this behavior after receiving a warning.
Even if I considered it unethical, I would still want this test to be performed, because I value kernel security above petty ideological concerns.
If this is illegal, then I don't think it should be illegal. There's always debates about the legality of hacking, but there's no doubt that many illegal (and arguably unethical) acts of hacking have improved computer security. If you remember the dire state of computer security in the early 2000s, remember that the solution was not throw all the hacker kids in jail.
The other part of the patch, that checks for `flow` being NULL may be unnecessary since it looks like the handle is always from an active context. But that's a guess. And it's only unreachable code.
The opinion I have from this is despite other patches being bad ideas, this one doesn't look like it. Because the other patches didn't make it past the mailing list, it demonstrates that the maintainers are doing a good enough job.
"A lot of these have already reached the stable trees. I can send you revert patches for stable by the end of today (if your scripts have not already done it)."
https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
If they don't, then that's the vulnerability in the process.
That's what they do every time.
It totally is if your goal as a hacker is generating a better outcome for security. Read the paper, see what they actually did, they just jerked themselves off over how they were better than the open source community, and generated a sum total of zero helpful recommendations.
So they subverted a process, introduced a Use After vulnerability and didn't do jack shit to improve it.
The beauty of it is that by "jerking themselves off", they are generating a better outcome for security. In spirit, this reaction of the kernel team is not that different from Microsoft attempting to bring asshole hacker kids behind bars for exposing them. When Microsoft realized that this didn't magically make Windows more secure, they fixed the actual problems. Windows security was a joke in the early 2000s, now it's arguably better than Linux. Why? Because those asshole hacker kids actually changed the process.
> So they subverted a process, introduced a Use After vulnerability and didn't do jack shit to improve it.
The value added here is to show that the process could be subverted, the lessons are to be learned by someone else.
If you show up to a kernel developer's house, put a gun to their head and tell them to approve the PR, that process can also be subverted...
But at the end of the day, sometimes there's just no way to do ethically do the experiment you want to do, and the right solution to that is to just live with being unable to do certain experiments.
The best they can do is notify the maintainers after they got the result for their research and give the maintainers an easy way to recover from vulnerability they intentionally create.
It is wasting a lot of peoples' time.
> What's the benefit? You raise trust in the process behind one of the most critical pieces of software.
I'm skeptical that a research paper by some nobodies from a state university will accomplish this.
If you run a test on your codebase and it passes, do you find that writing the test was a waste of time?
> I'm skeptical that a research paper by some nobodies from a state university will accomplish this.
It did for me.
> This is not ok, it is wasting our time, and we will have to report this, AGAIN, to your university...
> if you have a list of these that are already in the stable trees, that would be great to have revert patches, it would save me the extra effort these mess is causing us to have to do...
> Academic research should NOT waste the time of a community.
> The huge advantage of being "community" is that we don't need to do all the above and waste our time to fill some bureaucratic forms with unclear timelines and results.
Seems they don't think it is a good use of their time, no. But I'm sure you know a lot more about kernel development and open source maintenance than they do, right?
Wouldn't the correct term for that be: malicious threat actor?
Red team penetration testing doesn't involve the element of surprise, and is pre-arranged.
Intentionally wasting peoples time, and then going further to claim you weren't, is a malicious act as it intends to do harm.
I agree though, it's fascinating but only in the true crime sense.
Once any maintainer of the community responds to the email,indicating “looks good”,we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> If it quacks like a duck and waddles like a duck, then it is a duck.
A lot of horrible things have happened on the Internet by following that philosophy. I think it's imperative to learn the rigorous facts and different interpretations of them, or we will continue to great harm and be easily manipulated.
Seems more like low grade journalism to me.
The existing paper researched the feasibility of unknown actors to introduce vulnerable code. The hypothetical second paper has the same basis, but is from the vantage point of a known bad actor.
Reading through the mailing list (as best I can), the maintainer's response to the latest buggy patches seemed pretty civil[1] in general, and even more so considering the prior behavior. And the submitter's response to that (quoted here[2]) went to the extreme end of defensiveness. Instead of addressing or acknowledging anything in the maintainer's message, the submitter:
- Rejected the concerns of the maintainer as "wild accusations bordering on slander"
- Stating their naivety of the kernel code, establishing themselves as a newbie
- Called out the unfriendliness of the maintainers to newbies and non-expects
- Accused the maintainer of having preconceived biases
An empathetic reading of their response is that they really are a newbie trying to be helpful and got defensive after feeling attacked. But a cynical reading of their response is that they're attempting to exploiting high-visibility social issues to pressure or coerce the maintainers into accepting patches from a known bad actor.
The cynical interpretation is as much social-exploit-vector vulnerability research as what they did before. Considering how they deflected on the maintainer's concerns stemming from their prior behavior and immediately pulled a whole bunch of hot-button social issues into the conversation at the same time, the cynical interpretation seems at least plausible.
[1] https://lore.kernel.org/linux-nfs/YH5%2Fi7OvsjSmqADv@kroah.c...
[2] https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
It looks like actual security vulnerabilities were successfully added to the stable branch based on that comment.
Similarly, if a research paper shows that its hypothesis is false, the author might feel that it was a waste of time having worked on it, which can lead to publication bias.
To be clear, I don't think what the kernel maintainers did is wrong; it's just sad that all past and future potentially genuine contributions to the kernel from the university have been caught in the crossfire.
Sure they can only automatically ban the .edu address, but it would be pretty meaningless to just ban the university email host, but be ok with the same people submitting patches from personal accounts.
I would also explicitly ban every person involved with this "research" and add their names to a hypothetical ban list.
The reputational damage will be lasting, both for the researchers, and for UMN.
The biggest issue around this is consent. You can totally send an email saying "we're doing research on the security implications of the pull request process, can we send you a set of pull requests and you can give up approve/deny on each one?"
> If you remember the dire state of computer security in the early 2000s, remember that the solution was not throw all the hacker kids in jail.
You weren't there when Mirai caused havok due to thousands of insecure IoT devices getting pwned and turned into a botnet... and introducing more vulnerabilities is never the answer.
"because I value kernel security above petty ideological concerns"
This implies that this is the only or main way security is achieved. This is not true. Also, "valuing kernel security above other things"... is an ideological concern. You just happen to value this ideology more than other ideological concerns.
"whether something is ethical is an opinion"
It is, but there are bases for forming opinions on what is moral and ethical. In my opinion, secretly testing people is not ethical. Again, the difference here is consent. Plenty of organizations agree to probing/intrusion attempts; there is no reason to secretly do this. Again, security is not improved only by secret intrusion attempts.
"there's no doubt that many illegal (and arguably unethical) acts of hacking have improved computer security"
I don't believe in the ends justify the means argument. Either it's ethical or it isn't; whether or not security improved in the meantime is irrelevant. Security also improves in its own regard over time.
I do agree that the way the current laws regarding "hacking" are badly worded and very punitive, but crimes are crimes. Just because you like that hacking or think it may be beneficial does not change the fact that it was unauthorized access or an intentional attempt to submit bad, buggy code, etc.
We have to look at it exactly like we look at unauthorized access in i.e. business properties or peoples' homes. That doesn't change just because it's digital. You don't randomly walk up to your local business with a lock picking kit to "test their security". You don't randomly steal someone's wallet to "test their security". Why is the digital space any different?
Maybe that's what they claim to do, but how do you know for sure? How do you test for it?
> This implies that this is the only or main way security is achieved.
It doesn't, there are many facets of security, social engineering being one of them. Maybe it's controversial to test something that requires misleading people, but realistically the only alternative is to ignore the problem. I prefer not to do that.
> Plenty of organizations agree to probing/intrusion attempts; there is no reason to secretly do this.
Yes there is: Suppose you use some company's service and they refuse to cooperate in regards to pentesting: The "goody two-shoes" type of person just gives up. The "hacker type" puts on their grey hat and plays some golf. Is that unethical? What if they expose some massive flaw that affects millions of unwitting people?
> I don't believe in the ends justify the means argument.
Not all ends justify all means, but some ends do justify some means. In fact, if it's a justification to some means, it's almost certainly an end.
> I do agree that the way the current laws regarding "hacking" are badly worded and very punitive, but crimes are crimes.
Tautologically speaking, crimes are indeed crimes, but what are you trying to say here? Just because it's a crime doesn't mean it is unethical. Sometimes, not performing a crime is unethical.
> You don't randomly walk up to your local business with a lock picking kit to "test their security".
Yes, but only because that's illegal, not because it is unethical.
> You don't randomly steal someone's wallet to "test their security".
Again, there's nothing morally wrong with "stealing" someone's wallet and then giving it back to them. Better I do it than some pickpocket. I have been tempted on numerous occasions to do exactly that, but it's rather hard explaining yourself in such a situation...
> Why is the digital space any different?
Because the risk of running into a physical altercation is quite low, as is the risk of getting arrested.
Our society is built on trust. Do you test the water from the city every time you drink it? Etc. Days like today show that, yes, the kernel team is doing their job.
How about -you- prove that they -aren't- doing their job?
"Suppose you use some company's service and they refuse to cooperate in regards to pentesting ... Is that unethical?"
Yes. You are doing it without their consent. It is unethical. Just because you think you are morally justified in doing something without someone's consent does not mean that it is not unethical. Just because you think the overall end result will be good does not mean that the current action is ethical.
"Yes, but only because that's illegal, not because it is unethical."
This is very pedantic. It's both illegal and unethical. How would you like if it you had a business and random people came by and picked locks, etc, in the "name of security"? That makes zero sense. It's not your prerogative to make other people more secure. If they are insecure and don't want to test it, then it's their own fault when a malicious actor comes in.
"Again, there's nothing morally wrong with "stealing" someone's wallet and then giving it back to them"
Yes, it is morally wrong. In that scenario, you -are- the pickpocket. This is a serious boundary that is being crossed. You are not their parent. You are not their caretaker or guardian. You are not considering their consent -at all-. You have no right to "teach people lessons" just because you feel like you are okay with doing that. If you did that to me I would not hang out with you ever again, and let people know that you might randomly take their stuff or cross boundaries for "ideological reasons".
"Because the risk of running into a physical altercation is quite low, as is the risk of getting arrested. "
This is admission that you know what you're doing is wrong, and the only reason you do it digitally is because it's more difficult to receive consequences for it.
I strongly urge you to start considering consent of other people before taking actions. You can voice your concerns, but things like taking a wallet or picking a lock is crossing the line. Either they will take the advice or not, but you cannot force it by doing things like that.
The way these (intrusive) tests (e.g. anti phishing) are performed within organizations would be with the knowledge and a very strongly worded contract between the owners of the company and the party conducting the tests.
It is illegal in most of the world today. Even if you disagree with responsible disclosure you would be well advised not to send phishing mail to companies (whether your intention was to improve their security or not is beside the point).
That being said, I think it would've made more sense for them to have created some dummy complex project for a class and have say 80% of the class introduce "good code", 10% of the class review all code and 10% of the class introduce these "hypocrite" commits. That way you could do similar research without having to potentially break legit code in use.
I say this since the crux of what they're trying to discover is:
1. In OSS anyone can commit.
2. Though people are incentivized to reject bad code, complexities of modern projects make 100% rejection of bad code unlikely, if not impossible.
3. Malicious actors can take advantage of (1) and (2) to introduce code that does both good and bad things such that an objective of theirs is met (presumably putting in a back-door).
If leadership was on board, they could have then proceeded with the test under the supervision of those core maintainers who ensure introduced security holes don't find their way into stable. The insiders themselves would abstain from reviewing those patches to see if review by others catches them.
If leadership was not on board, they should have respected the wishes of the Linux team and found another high-visibility open-source project who is more amenable to the project. There are lots of big open-source projects to choose from, the kernel simply happens to be high-profile.
I still think the best thing for them would be to simply create their own project and force their own students to commit, but they probably felt that doing that would be too contrived.
they could've done the much harder work of studying all of the incoming patches looking for bugs, and then just not reporting their findings until the kernel team accepts the patch.
the kernel has a steady stream of incoming patches, and surely a number of bugs in them to work with.
yeah it would've cost more, but would've also generated significant value for the kernel.
That's a really stupid behavior ...
In closed source, nobody would even check. Modern DevOps has essentially replaced manual code review with unit tests.
Secondly, the researcher’s attitude sounds high and mighty - making process improvement suggestions when their own ethical compass is in question. Their “experiment” was “what would happen if...”. Well, bans happen. If one starts a fight don’t get indignant over a bloody nose, lol
Ps: I have put security researcher in quotes because this kind of thing is not security research, it's a publicity stunt.
No they weren't. They made sure the bad code never made it in. They are only guilty of wasting peoples time.
How about you think about what they just proved, about the actors that *actually* try to break the security of the kernel.
> 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
..which is generally a good thing even if it also protects clearly malicious actions like this.
Is there not some sort of equivalent in this field?
That'll be a fraud, no?
...and yet FOSS and especially Linux is very widely used in military devices including weapons.
Because it's known to be less insecure than most alternatives.
As if there is some other software that is "military-grade" by the same measure? What definition are you using for that term, anyway?
A lot of people claim that there's a lot of eyes on the code and thus introducing vulnerabilities is unlikely. This research clearly has bruised some egos bad.
Eric Raymond claimed so, and a lot of people repeated his claim, but I don't think this is the same thing as "a lot of people claim" -- and even if a lot of people claim something that is obviously stupid, it doesn't make the thing less obviously stupid, it just means it's less obvious to some people for some reasons.
And they are correct. Unfortunately sometimes the number of eyes is not enough.
The alternative is closed source, which has prove to be orders of magnitude worse, on many occasions.
Greg: You can't quit, you're fired.
Imagine, saying we would like to test how fire department responds to fire, by setting buildings on fire in NYC.
Also the buildings are not random, but safety critical infrastructure, but this is good, you can advise later:'put a "please do not ignite" sign on the building'.
In a network security analogy, this is just unsolicited hacking VS being a penetration test which it claims more so to be.
The paper doesn't actually have concrete suggestions for tools, just hand-waving about "use static analysis tools, better than the ones you already use" and "use fuzzers, better than those that already exist."
The work was a stunt to draw attention to the problem of malicious committers. In that regard, it was perhaps successful. The authors' first recommendation is for the kernel community to increase accountability and liability for malicious committers, and GregKH is doing a fantastic job at that by holding umn.edu accountable.
vvv CID 1503716: Null pointer dereferences (REVERSE_INULL) vvv Null-checking "rm" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
and tools are useful, but given the resources and the know-how of those who compete in the IOCC I think we'd have to assume they'd be able to get something through. It'd have an even higher chance of success if it could be built to target a particular hardware combination (of a desired victim) as you could make the exploit dependent on multiple parts of the code (and likely nobody would ever determine the extent, as they'd find parts of it and fix them independently).
Obviously, bugs gets introduced into all software projects all the time. And the bugs don't know whether they've been put there intentionally or accidentally. Alls bugs that ever appeared in the linux kernel obviously made it through the review process. Even when no-one actively tried to introduce them.
So, why should it not be possible to intentionally insert bugs if it already "works" unintentionally? What is the insight gained from this innovative "research"?
Responding properly to that statement would require someone to step out of the HN community guidelines.
Social shame and reputation damage may be useful defense mechanisms in general, but in a hacker culture where the right to make up arbitrarily many secret identities is a moral imperative, people who burn their identities can just get new ones. Banning or shaming is not going to work against someone with actual malicious intent.
Inserting backdoors in the form of bugs is not difficult. Just hijack the machine of a maintainer, insert a well placed semicolon, done!
Do you remember the quote of Linus Torvalds ? "Given enough eye balls, all bugs are shallow." ? Do you really believe the Linux source code is being reviewed for bugs?
By the way, how do you write tests for a kernel?
I like open source, but security implies a lot of different problems and open source is not always better for security.
"Plainly put, the patch demonstrates either complete lack of understanding or somebody not acting in good faith. If it's the latter[1], may I suggest the esteemed sociologists to fuck off and stop testing the reviewers with deliberately spewed excrements?"
https://lore.kernel.org/lkml/YH4Aa1zFAWkITsNK@zeniv-ca.linux...
vvv CID 1503716: Null pointer dereferences (REVERSE_INULL)
vvv Null-checking "rm" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.I mean, sneakily introducing vulnerabilities obviously only works if you don't start your messages by announcing you are one of the guys known to be trying to do so...
As Joe Blow at home who happens to go to school or work there you could submit even if you were part of the research team. Because you are not representing the university. The university is banned.
That there's an ethical way of testing processes which includes asking for permission and using proven tested methods like sending a certain amount of items N where X are compromised and Y are not compromised and seeing the ratio of K where K are rejected items and the ratio of rejected items which are compromised K/X versus non-compromised items K/Y.
By breaking the ethical component, the entire scientific method of this paper is broken... now I have to go check the kernel pull requests list to see if they sent 300 pull requests and got one accepted or if it was a 1:1 ratio.
Again, that's not the same test. You are introducing bias. You are not observing the same thing. Maybe you think that observation is of equal value, but I don't.
> By breaking the ethical component, the entire scientific method of this paper is broken...
Not at all. The scientific method is amoral. The absolute highest quality of data could only be obtained by performing experiments that would make Joseph Mengele faint.
There's always an ethical balance to be struck. For example, it's not ethical to perform experiments on rats to develop insights that are of no benefit to these rats, nor the broader rat population. If we applied our human ethical standards to animals, we could barely figure anything out. So what do we do? We accept the trade-off. Ethical concerns are not the be-all-end-all.
In this case, I'm more than happy to have the kernel developers be the labrats. I think the tradeoff is worth it. Feel free to disagree, but I consider the ethical argument to be nothing but hot air.
Доверяй, но проверяй
> Do you test the water from the city every time you drink it?
Not every time, but on a regular basis.
> Days like today show that, yes, the kernel team is doing their job.
...and I am happy to report that my water test results did not raise concerns.
> Yes. You are doing it without their consent. It is unethical.
I disagree that it is unethical just because it lacks consent. Whistleblowing also implies that there is no consent, yet it is considered ethical. Suppose that Facebook leaves private data out in the open, then refuses to allow anyone to test their system for such vulnerabilities. It would be unethical not to ignore their consent in this regard.
> How would you like if it you had a business and random people came by and picked locks, etc, in the "name of security"? That makes zero sense.
I would find it annoying, of course. Computer hackers are annoying. It's not fun to be confronted with flaws.
The thing is, security is not about how I feel. We need to look at things in proportion. If my business was a random shoe store, then perhaps it doesn't matter that my locks aren't that great, perhaps these lockpickers are idiots. If my business houses critical files that absolutely must not be tampered with, then I can not afford to have shitty locks and frankly I should be grateful that someone is testing them, for free.
> Yes, it is morally wrong. In that scenario, you -are- the pickpocket. This is a serious boundary that is being crossed. You are not their parent. You are not their caretaker or guardian...
Can we just agree to disagree on morals?
> This is admission that you know what you're doing is wrong, and the only reason you do it digitally is because it's more difficult to receive consequences for it.
Not at all, those are two entirely separate things. I wouldn't proclaim my atheism in public while visiting Saudi Arabia - not because I think there's anything morally wrong with that, but because I don't want the trouble.
> I strongly urge you to start considering consent of other people before taking actions.
You use "consent" as if it was some magical bane word in every context. In reality, there's always a debate to be had on what should and should not require consent. For example, you just assumed my consent when you quoted my words, yet I have never given it to you.
What HN would call "ancient".
A university or corporate e-mail address helps with the former: even if the individual doesn't put their real name into their email address, the institution still maintains that mapping. The possibility of professional, legal, or social consequences attaching to your real-world identity (as is likely to happen here) is a generally-effective deterrent.
Raymond doesn't seem to claim anything like "there are sufficient eyes to swat all bugs in the kernel", or "there are eyes on all parts of the code", or "'bugs' covers all possible security flaws", or etc. He particularly mentions uptime and crashing, so less charitably the statement is "there are no crashing or corruption bugs so deep that a large enough quantity of volunteers can't bodge some way past them". Which leaves plenty of room for less used subsystems to have nobody touching them if they don't cause problems, patches that fix stability at the expense of security, absense of careful design in some areas, the amount of eyes needed being substantially larger than the amount of eyes involved or available, that maliciously submitted patches are different from traditional bugs, and more.
The requirements arising from US Govt grant funding may well be more strict than UMN.
I'm not sure how the law works in such cases, but surely the IRB would eventually have to realize that an explicit denouncement by the victims means that the "research" cannot go ahead
Eg - If you want to do kernel related research, don’t go to the university of Minnesota.
- Human subjects - Intentionally misleading/misrepresenting things, potential for a lot of damage, given how widespread Linux is - No informed consent at all!
Sorry but one cannot use unsuspecting people as guinea pigs for research, even if it is someone from a reputable institution.
You don't test a bank or Fortune 500 security system without buy-in of leadership ahead of time.
In any case as I mentioned before I disagree with what they did.
> We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our correct patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. All the UAF-introducing patches stayed only in the email exchanges, without even becoming a Git commit in Linux branches. Therefore, we ensured that none of our introduced UAF bugs was ever merged into any branch of the Linux kernel, and none of the Linux users would be affected.
It seems that the research in this paper has been done properly.
EDIT: since several comments come to the same point, I paste here an observation.
They answer to these objections as well. Same section:
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
And, coming to ethics:
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
When I fill out the NIH's "is this human research" tool with my understanding of what the study did, it tells me it IS human subjects research, and is not exempt. There was an interaction with humans for the collection of data (observation of behavior), and the subjects haven't prospectively agreed to the intervention, and none of the other very narrow exceptions apply.
How is wasting the time of maintainers of one of the most popular open source project "done properly"?
Also, someone correct me if I'm wrong, but I think if you do experiments that involve other humans, you need to have their consent _before_ starting the experiment, otherwise you're breaking a bunch of rules around ethics.
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
And, coming to ethics:
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
I would have reacted the same way Greg did - I don't care what credentials someone has or what their hidden purpose is, if you are intentionally submitting malicious code, I would ban you and shame you.
If particular researchers continue to use methods like this, I think they will find their post-graduate careers limited by the reputation they're already establishing for themselves.
We can independently conclude this kind of research has put open source projects in danger by getting vulnerabilities that could carry serious real world consequences. I could imagine many other ways to carrying out this experiment without the consequences it appears to have had, like perhaps inviting developers to a private repository and keeping the patch from going public, or collaborating with maintainers to set up a more controlled experiment without risks.
This seems by all appearances an unilateral and egoistic behavior without great thought into its real world consequences.
Hopefully researchers learn from it and it doesn't discourage future ethical kernel research.
Even if none of the patches made into the kernel (which doesn't seem to be true, according to other accounts), it's still possible to do permanent damage to the community of kernel maintainers.
Essentially, the researchers were not in control to stop the experiment if it deviated from expectations. They were relying on the exact system they were testing to trigger its halt.
We also don't know what details they gave the IRB. They may have passed through due to IRB's naivete on this: It had a high human component because it was humans making many decisions in this process. In particular, there was the potential to cause maintainers personal embarrassment or professional censure by letting through a bugged patch. If the researchers even considered this possibility, I doubt the IRB would have approved this experimental protocol if laid out in those terms.
"In the past several years, we devote most of our time to improving the Linux kernel, and we have found and fixed more than one thousand kernel bugs"
But someone upthread posted that this group has a total of about 280 commits in the kernel tree. That doesn't seem like anywhere near enough to fix more than a thousand bugs.
Also, the clarification then says:
"the extensive bug finding and fixing experience also allowed us to observe issues with the patching process and motivated us to improve it"
And the way you do that is to tell the Linux kernel maintainers about the issues you observed and discuss with them ways to fix them. But of course that's not at all what this group did. So no, I don't agree that this research was done "properly". It shouldn't have been done at all the way it was done.
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
IEEE is just the publishing organisation and doesn't review research. That's handled by the program committee that each IEEE conference has. These committees consist of several dozen researchers from various institutions that review each paper submission. A typical paper is reviewed by 2-5 people and the idea is that these reviewers can catch ethical problems. As you may expect, there's wide variance in how well this works.
While problematic research still slips through the cracks, the field as a whole is getting more sensitive to ethical issues. Part of the problem is that we don't yet have well-defined processes and expectations for how to deal with these issues. People often expect IRBs to make a judgement call on ethics but many (if not most) IRBs don't have computer scientists that are able to understand the nuances of a given research projects and are therefore ill-equipped to reason about the implications.
That doesn't stop someone from lying about it, but it's not a casual claim, and doing so would probably bring community censure (as well as being easily falsifiable after time).
Security research is not always the most ethical branch of computer science, to say it mildly. Those are the people selling exploits to oppressive regimes, allowing companies to sit on "responsibly reported" bugs for years while hand-wringing about "that wasn't in the attacker model, sorry our 'secure whatever' we sold is practically useless". Of course the overall community isn't like that, but the bad apples spoil the bunch. And the aforementioned unethical behaviour even seems widely accepted.
What this research demonstrates is that you can quite easily slip back doors into an open contribution (which is often but not always associated with open source) project with supposedly the most eyes on it. That’s not true for any closed source project which is definitely not open contribution. (You can go for an open source supply chain attack, but that’s again a problem for open source.)
> What this research demonstrates is that you can quite easily slip back doors into an open contribution
To make a fair comparison you should contrast it with companies or employees placing a backdoors into their own closed source software.
It's extremely easy to do and equally difficult to spot for end users.
Rogue companies/employees is really a different security problem that’s not directly comparable to drive-by patches (the closest comparison is a rogue open source maintainer).
[1] https://twitter.com/SarahJamieLewis/status/13848713855379087...
It had a high human component because it was humans making decisions in this process. In particular, there was the potential to cause maintainers personal embarrassment or professional censure by letting through a bugged patch. If the researchers even considered this possibility, I doubt the IRB would have approved this experimental protocol if laid out in those terms.
The only relevant question is: "Will the investigator use ... information ... obtained through ... manipulations of those individuals or their environment for research purposes?"
which could be idly thought of as "I'm just sending an email, what's wrong with that? That's not manipulating their environment".
But I feel they're wrong.
https://grants.nih.gov/policy/humansubjects/hs-decision.htm would seem to agree that it's non-exempt (i.e. potentially problematic) human research if "there will be an interaction with subjects for the collection of ... data (including ... observation of behaviour)" and there's not a well-worn path (survey/public observation only/academic setting/subject agrees to study) with additional criteria.
Not to excuse them at all, I think the results are entirely appropriate. What they're seeing is the immune system doing its job. Going easy on them just because they're a university would skew the results of the research, and we wouldn't want that.
One of the important rules you must agree to is that you cannot deceive anyone in any way, no matter how small, if you are going to claim that you are doing exempt research.
These researchers violated the rules of their IRB. Someone should contact their IRB and tell them.
If the IRB approved this as exempt and they had an accurate understanding of the experiment, it makes me question the IRB itself. Whether the researchers were dishonest with the IRB or the IRB approved this as exempt, it's outrageous.
Besides, all we have to do is look at the outcome: Outrage on the part of the organization targeted, and a ban by that organization that will limit the researcher's institution from conducting certain types of research.
If this human-level harm was the actual outcome means the experiment was a de fact experiment including human subjects.
I do recommend participating more in other threads and a little less in this thread, where you're repeating pretty much the same point over and over.
Or you have a closed-source component you bought from someone who pinky-swears to be following secure coding practices and that their code is of course bug-free...
And that's why nation-state attackers do it routinely.
For the end user, the threat model is about the presence of a malicious function in some binary.
Regardless if the developers are an informal community, a company, a group of companies, an NGO. They are all "outside" to the end user.
Closed source software (e.g. phone apps) breach user's trust constantly, e.g. with privacy breaching telemetries, weak security and so on.
If Microsoft weakens encryption under pressure from NSA is it "inside" or "outside"? What matters to end users is the end result.
There's absolutely no reason everyone's threat model has to equate insiders with outsiders. If a stranger on the street gives you candy, you'll probably check it twice or toss it away out of caution. If a friend or family member does the same thing, you'll probably trust them and eat it. Obviously at the end of the day, your concern is the same: you not getting poisoned. That doesn't mean you can (or should...) treat your loved ones like they're strangers. It's outright insane for most people to live in that manner.
Same thing applies to other things in life, including computers. Most people have some root of trust, and that usually includes their vendors. There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally. Most people, in fact, should not do so. (And this should not be a controversial position...)
1) Unless you exclusively run software written by close friends both Linux and $ClosedOSCompany are equally "outsiders"
2) I regularly trust strangers to make medicines I ingest any fly airplanes I'm on. I would not trust any person I know to fly the plane because they don't have the required training.
So, trust is not so simple, and that's why risk analysis takes time.
> There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally
...and that's why plenty of critical system around the world, including weapons, run on Linux and BSD, especially around countries that don't have the best relations with US.
Edit: All conference are different, I dont know if it applies to that one.
Not sure how that passage justifies wasting the time of these people working on the kernel. Because the issues they pretend to fix are real issues and once their research is done, they also submit the fixes? What about the patches they submitted (like https://lore.kernel.org/linux-nfs/20210407001658.2208535-1-p...) that didn't make any sense and didn't actually change anything?
> And, coming to ethics:
So it seems that they didn't even just mislead the developers of the kernel, but they also misled the IRB board, as they would never approve it without getting consent from the developers since they are experimenting on humans and that requires consent.
Even in the section you put above, they even confess they need to interact with the developers ("this experiment will take certain time of maintainers in reviewing the patches"), so how can they be IRB-exempt?
The closer you look, the more sour this whole thing smells.
I was wondering why he banned the whole university and not just these particular researchers. I think your quote is the answer to that. I'm not sure on what basis this exemption was granted.
Here's what the NIH says about it:
Definition of Human Subjects Research
https://grants.nih.gov/policy/humansubjects/research.htm
Decision Tool: Am I Doing Human Subjects Research?
https://grants.nih.gov/policy/humansubjects/hs-decision.htm
And even if they did find some way to justify it under their own rules, some of the research subjects clearly disagree.
https://lore.kernel.org/lkml/20210421130105.1226686-8-gregkh... for the full list
Inasmuch as the IRB marked this as "not human research" they appear to have erred.
I would be livid if I found that code from these "researchers" was running in a medical device that a family member relied upon.
And for some insane reason, they decided to test if these kinds of bugs would be caught by inventing some and just submitting the patches, without informing anyone beforehand.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
Or to prove its overall insecurity.
Insofar as this specific method of injecting flaws matches a foreign country's work done on U.S. soil - as many people in this thread have speculated - do people here think that U.S. three letter agencies (in particular NSA/CIA) should have the ability to look at whether the researchers are foreign agents/spies, even though the researchers are operating from the United States? For example, should the three letter agencies have the ability to review these researchers' private correspondence and social graphs?
Insofar as those agencies should have this ability, then, when should they use it? If they do use it, and find that someone is a foreign agent, in what way and with whom should they share their conclusions?
Also, while their assumption is interesting, there sure had to be an ethical and safe way to conduct this. Especially without allowing their bugs to slip into release.
From what I understand, this answer seems to be a "yes".
Of course, it is understandable that GKH is frustrated, and if his community do not like someone pointing out this issue, it is OK too.
However, one researcher does not represent the whole university, so it seems immature to vent this to other unrelated people just because you can.
Besides, how to test the idea without doing what they did? Can you show us a way?
As far as it's known, garbage code was not introduced into kernel.It was caught in the review process literally on the same day.
However, there has been merged code from the same people, which is not necessarily vulnerable. As a precaution the older commits are also being reverted, as these people have been identified as bad actors
That means that the researchers got bogus code into the kernel, got it accepted, and then said nothing for two weeks as the bogus commit spread through the Linux development process and ended up in the stable tree, and, potentially, in forks.
I think the Linux Foundation should make an example of this.
Sorry for being the paranoid one here, but reading this raises a lot of warning flags.
Static analysis is being done[1][2], in addition, there are also CI test farms[3][4], fuzzing farms[5], etc. Linux is a project that enough large companies have a stake in that there are some willing to throw resources like this at it.
Human review is supposed to be done through the mailing list submission process. How well this works depends in my experience from ML to ML.
[1] https://www.kernel.org/doc/html/v4.15/dev-tools/coccinelle.h...
[2] https://scan.coverity.com/projects/linux
Moving to rust to limit the scope of possible bugs.
Maybe not being nice is part of the immune system of open source.
How can one be so short-sighted?...
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
And similarly to U Minn, their IRB covered for them: https://lobste.rs/s/3qgyzp/they_introduce_kernel_bugs_on_pur...
My experience felt really shitty, and I'm sorry to see I'm not alone. If anyone is organizing a broad response to redress previous abuses or prevent future abuse, I'd appreciate hearing about it, my email's on my profile.
But, allow me to pull a different thread. How liable is the professor, the IRB, and the university if there is any calamity caused by the known code?
What is the high level difference between their action, and spreading malware intentionally?
[edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]
QFT.
Defund federal student loans. Make these universities stand on their own two feet or be replaced by something better.
Talk about predictive power in a hypothesis.
Getting banned from contributing is a light penalty.
There is absolutely zero evidence of this. None. In my opinion it's baseless speculation.
It's far more likely that they are upset over being called out, and are out of touch with regards as to what is ethical testing.
"Well if you're gonna be a jerk about it, I won't be sending any more patches."
If not you get cluttered up with bad code and people there for the experience. Like how stackoverflow is lost to rule zealots there for the game not for the purpose.
Something big and important should be intimidating and isn’t a public service babysitter...
If I have a community, bombarded by a random number of transient bad actors at random times, then if R0 > some threshold, my community inevitably trends to a cesspool, as each bad actor creates more negative members.
If I take steps to decrease R0, one of which may indeed be "blunt- and harshness to new contributors", then my community may survive in the face of equivalent pressures.
It's a valid point, and seems to have historical support via evidence of many egalitarian / welcoming communities collapsing due to the accumulation of bad faith participants.
The key distinction is probably "Are you being blunt / harsh in the service of the primary goal, or ancillary to the mission?"
[0] https://en.m.wikipedia.org/wiki/Basic_reproduction_number
You don't need to do that by telling them they're garbage. You can do it by getting them invested in growth and improvement.
Like?
Someone for whom being a bad actor is a day job will not get deterred by being told to fuck off.
Being nasty might deter some low key negative contributors - maybe someone who overestimates their ability or someone "too smart to follow the rules". But it might also deter someone who could become a good contributor.
If you ran a bank and had a bunch of rude bank tellers, you are only going to dissuade customers, not bank robbers.
Sustaining the belief that every submitter is an earnest, good, and altruistic person is painfully expensive and a waste of very valuable minds. Unhinged is unhinged and that needs to be managed, but keeping up the farce that there is some imaginary universe where the submitter is not wasting your time and working the process is wrong.
I see this in architecture all the time, where people feign ignorance and appeal to this idea you are obligated to keep up the pretense that they aren't being sneaky. Competent people hold each other accountable. If you can afford civility, absolutely use it, but when people attempt to tax your civility, impose a cost. It's the difference between being civil and harmless.
This is just a form of "well I'll just take my business elsewhere!". Chances are he'll try again under a pseudonym.
EDIT: University is fair game too.
The idea that "not being nice" is necessary is plainly ridiculous, but this post is pretty wild--effectively you're implying that they're just amateurs or something and that this is a novel idea nobody's considered, while billions and billions of dollars of business run atop Linux-powered systems.
What they don't do is hand over CI resources to randos submitting patches. That's why kernel developers receive and process those patches.
> This is not ok, it is wasting our time, and we will have to report this, AGAIN, to your university...
We have public funded rules (typically derived or pressured by availability of federal or state monies/resources) which are quite strict, have ethics and IRB boards, cover even behavioral studies like this where no direct physical harm is induced but still manipulates peoples' behaviors. This is the type of experiment you're referring to where you can't experiment on people without their consent (and by the way, I agree with this opinion).
Meanwhile, we have private funded research which has a far looser set of constraints and falls into everyday regulations. You can't really physically harm someone or inject syphilis in them (Tuskegee experiments) which makes sense, but when we start talking about human subjects in terms of data, privacy of data, or behavioral manipulation most regulation goes out the window.
These people likely could be reprimanded, even fired, and scarlet lettered making their career going forward more difficult (maybe not so much in this specific case because it's really not that harmful) but enough to screw them over financially and potentially in terms of career growth.
Meanwhile, some massive business could do this with their own funding and not bat an eye. Facebook could do this (I don't know why they would) but they could. Facebook is a prime example of largely unregulated human subject experimentation though. Social networks are a hotbed for data, interactions, and setting up experimentation. It's not just Facebook though (they're an obvious easy target), it's slews of businesses collecting data and manipulating it around consumers: marketing/advertising, product design/UX focusing on 'engagement', and all sorts of stuff. Every industry does this and that sort of human subject experimentation is accepted because $money$. Meanwhile, researchers from public funding sources are crucified for similar behaviors.
I'm not defending this sort of human subject experimentation, it's ethically questionable, wrong, and should involve punishment. I am however continually disgusted by the double standard we have. If we as a society really think this sort of experimentation on human subjects or human subject data is so awful, why do we allow it to occur under private capital and leave it largely unregulated?
Linux Bug fixes are open to the public. The experiment isn't on people but on bugs. I would be like filing different customer support complaints to change the behavior of a company -- you're not experimenting on people but the process of how that company interfaces with the public.
I see no wrong here including the Linux maintainers banning submissions from UoM which is completely justified as time wasting.
edit: oh, it seems they got an exemption, because it's software research - https://news.ycombinator.com/item?id=26890084 :|
And you're right, bugs in the linux kernel could have serious consequences.
[1] https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Maybe they had some plan for immediate revert when the bogus patch got into stable, but some people update stable quickly, for a good reason, and it's just not good to do this research this way.
See JSON.org License which says it "shall be used for Good, not Evil" and is not considered free software.
Nobody would continue to use linux if they randomly banned people from using it, regardless of the reason.
[side note] This is why I despise the term "open source". It obscures the important part of user freedom. The term "Free/libre software" is not perfect, but it doesn't obscure this.
also here we are seeing persons are having no interest in "growth and improvement", they are not even creating the good faith contributions to project.
Typically, OSS is both definitions at the same time - free monetarily, and "free" as in "freedom" to use. JSON is an interesting case of "free" monetarily but not totally "free for use".
So answering that they hope to get more material for papers, which is the only goal of researchers (and their main KPI), is quite deeper an answer than the question required.
It is pretty comfortably not sabotage under 18 USC 105, which requires proving intent to harm the national defense of the United States. Absent finding an email from one of the researchers saying "this use-after-free is gonna fuck up the tankz," intent would otherwise be nearly impossible to prove.
Presumably, this reference is intended to be to 18 USC ch. 105 (18 USC §§ 2151-2156). However, the characterization of required intent is inaccurate; the most relevant provision (18 USC § 2154) doesn’t require intent if the defendant has “reason to believe that his act may injure, interfere with, or obstruct the United States or any associate nation in preparing for or carrying on the war or defense activities” (emphasis added) during either a war or declared national emergency.
It wouldn’t take much more than showing evidence that the defendant was aware (or even was in a position likely to be exposed to information that would make him aware) that Linux is used somewhere in the defense and national security establishment to support the mental state aspect of the offense.
People introducing bad code on purpose, for a social experiment, are on a whole new level of bad and so would his anger be.
Could you provide references to some of this historical support?
Some can tolerate a steady influx of bad actors: some fall apart. There's probably a valid paper in there somewhere.
>>>On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits Qiushi Wu, and Kangjie Lu. To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
If they weren't doing it, then quality of SO would decrease for all of us.
It's in our interest to have strict mods on SO
However, in this particular case, I agree that it is not a valid argument since it is doubtful whether the beginning kernel contributor's patches are in good faith.
[1] Torvalds encouraging new contributors to send in trivial patches back in 2004: https://lkml.org/lkml/2004/12/20/255
It's like "be kind I'm new to the concept of airplanes, let me fly this airplane with hundreds of passengers"
From TFA: “The UMN had worked on a research paper dubbed "On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits". Obviously, the "Open-Source Software" (OSS) here is indicating the Linux kernel and the University had stealthily introduced Use-After-Free (UAF) vulnerability to test the susceptibility of Linux.”
Suggesting a foreign government could be leaping to conclusions as well, given domestic activists with an agenda may do the same thing. A linux kernel backdoor is valuable to a lot of different interests. Hence why counter-intelligence should be involved.
However, I just looked at the names of the people involved and I don't know. Even if they were Taiwanese, that's an allied country, so I wouldn't expect it. Who were you thinking of?
(Not to say I like the StackExchange community much. It's far too top-down directed for me. But I'm very much sympathetic to the spirit of strict moderation.)
There are all sorts of community websites around the world. Which have developed into a serious SO contendor? IMO many things are threatening SO’s relevance, but they don’t look anything like it, which suggests that what SO is doing wrong isn’t the small details.
For example, I’d argue that Discord has become the next place for beginners to get answers, but chat rooms are very different from SO. For one thing, the help is better because someone else can spend their brain power to massage your problem. And another is that knowledge dies almost instantly.
Forgive me if I wasn't being clear. It seems like your core point is that SO's rules are on the whole good for keeping it focused, and it seems like you are assuming I'm a beginner programmer who is frustrated with SO for not being more beginner friendly and thus advising me on what I should do instead. I feel like you are shadow boxing a little.
I think we probably mostly agree; I think SO gets an unfortunate reputation as a good place for beginners (as opposed to a sort of curated wiki of asked and answered questions on a topic, a data store of wisdom), and that in general beginners are probably best served by smaller 1-1 intervention. I usually suggest people seek out a mentor, it had never occurred to me that Discord could be a good way to go about this.
The original point I was trying to make is simply that you can see overzealous rule following on SO and that a form of that is in inappropriately closed as duplicate questions.
https://news.ycombinator.com/item?id=26887670&p=2
https://news.ycombinator.com/item?id=26887670&p=3
https://news.ycombinator.com/item?id=26887670&p=4
https://news.ycombinator.com/item?id=26887670&p=5
https://news.ycombinator.com/item?id=26887670&p=6
https://news.ycombinator.com/item?id=26887670&p=7
(Posts like this will go away once we turn off pagination. It's a workaround for performance, which we're working on fixing.)
Also, https://www.neowin.net/news/linux-bans-university-of-minneso... gives a bit of an overview. (It was posted at https://news.ycombinator.com/item?id=26889677, but we've merged that thread hither.)
Edit: related ongoing thread: UMN CS&E Statement on Linux Kernel Research - https://news.ycombinator.com/item?id=26895510 - April 2021 (205 comments and counting)
"We experimented on the linux kernel team to see what would happen. Our non-double-blind test of 1 FOSS maintenance group has produced the following result: We get banned and our entire university gets dragged through the muck 100% of the time".
That'll be a fun paper to write, no doubt.
Additional context:
* One of the committers of these faulty patches, Aditya Pakki, writes a reply taking offense at the 'slander' and indicating that the commit was in good faith[1].
Greg KH then immediately calls bullshit on this, and then proceeds to ban the entire university from making commits [2].
The thread then gets down to business and starts coordinating revert patches for everything committed by University of Minnesota email addresses.
As was noted, this obviously has a bunch of collateral damage, but such drastic measures seem like a balanced response, considering that this university decided to _experiment_ on the kernel team and then lie about it when confronted (presumably, that lie is simply continuing their experiment of 'what would someone intentionally trying to add malicious code to the kernel do')?
* Abhi Shelat also chimes in with links to UMN's Institutional Review Board along with documentation on the UMN policies for ethical review. [3]
[1]: Message has since been deleted, so I'm going by the content of it as quoted in Greg KH's followup, see footnote 2
[2]: https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
[3]: https://lore.kernel.org/linux-nfs/3B9A54F7-6A61-4A34-9EAC-95...
I also now have submitted a patch series that reverts the majority of all of their contributions so that we can go and properly review them at a later point in time: https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
As an OSS maintainer (Node.js and a bunch of popular JS libs with millions of weekly downloads) - I feel how _tempting_ it is to trust people and assume good faith. Often since people took the time to contribute you want to be "on their side" and help them "make it".
Identifying and then standing up to bad-faith actors is extremely important and thankless work. Especially ones that apparently seem to think it's fine to experiment on humans without consent.
So thanks. Keep it up.
From a different thread: https://lore.kernel.org/linux-nfs/CADVatmNgU7t-Co84tSS6VW=3N... > A lot of these have already reached the stable trees.
Apologies in advance if my questions are off the mark, but what does this mean in practice?
1. If UNM hadn't brought any attention to these, would they have been caught, or would they have eventually wound up in distros? 'stable' is the "production" branch?
2. What are the implications of this? Is it possible that other malicious actors have done things like this without being caught?
3. Will there be a post-mortem for this attack/attempted attack?
What a joke - not sure how they can rationalize this as valuable behavior.
Also to assume _all_ commits made by UMN, beyond what's been disclosed in the paper, are malicious feels a bit like an overreaction.
I'm currently wondering how much of these patches could've been flagged in an automated manner, in the sense of fuzzing specific parts that have been modified (and a fuzzer that is memory/binary aware).
Would a project like this be unfeasible due to the sheer amount of commits/day?
Are you not concerned these malicious "researches" will simply start using throwaway gmail addresses?
Since this researcher is apparently not an established figure in the kernel community, my expectation is the patches have gone through the most rigorous review process. If you think the risk of malicious patches from this person have got in is high, it means that an unknown attacker deliberately concerting complex kernel loop hole would have a even higher chance got patches in.
While I think the researcher's actions are out of line for sure. This "I will ban you and revert all your stuff" retaliation seems emotional overaction.
THANK YOU! After reading the email chain, I have a much greater appreciation for the work you do for the community!
Just reverting those patches (which may well be correct) makes no sense, you and/or other maintainers need to properly review them after your previous abject failure at doing so, and properly determine whether they are correct or not, and if they aren't how they got merged anyway and how you will stop this happening again.
Or I suppose step down as maintainers, which may be appropriate after a fiasco of this magnitude.
I hope we hear from the IRB in about a year stating exactly what happened. Real investigations of bad conduct should take time to complete correctly and I want them to do their job correctly so I'll give them that time. (there is the possibility that these are good faith patches and someone in the linux community just hates this person - seems unlikely but until a proper independent investigation is done I'll leave that open.)
https://raw.githubusercontent.com/QiushiWu/qiushiwu.github.i...
> We send the emails to the Linux communityand seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter. The experiment will not collect any personal data, individual behaviors, or personal opinions. It is limited to studying the patching process OSS communities follow, instead of individuals.
First of all, this is completely irresponsible, what if the patches would've made their way into a real-life device? The paper does mention a process through which they tried to ensure that doesn't happen, but it's pretty finicky. It's one missed email or one bad timezone mismatch away from releasing the kraken.
Then playing the slander victim card is outright stupid, it hurts the credibility of actual victims.
The mandate of IRBs in the US is pretty weird but the debate about whether this was "human subject research" or not is silly, there are many other ethical and legal requirements to academic research besides Title 45.
That'd be great, yup. And the linux kernel team should then strongly consider undoing the blanket ban, but not until this investigation occurs.
Interestingly, if all that happens, that _would_ be an intriguing data point in research on how FOSS teams deal with malicious intent, heh.
I think the real problem is rooted more fundamentally in academia than it seems. And I think it has mostly to do with a lack of ethics!
We presented students with an education protocol designed to make a blind subset of them fail tests. Then measured if they failed the test to see if they independently learned the true meaning of the information.
Under any sane IRB you would need consent of the students. This is failure on so many levels.
(edit to fix typo)
Has anyone from the "research" team commented and confirmed this was even them or a part of their research? It seems like the only defense is from people who did google-fu for a potentially outdated paper. At this point we can't even be sure if this isn't a genuinely malicious actor using comprimised credentials to introduce vulnerabilities.
Conclusions: As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials.Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.
With the footnote: Contributors: GCSS had the original idea. JPP tried to talk him out of it. JPP did the first literature search but GCSS lost it. GCSS drafted the manuscript but JPP deleted all the best jokes. GCSS is the guarantor, and JPP says it serves him right
People got swatted for less.
As heard frequently on ASP, along with "Room Temperature Challenge."
"The University of Minnesota Department of Computer Science & Engineering takes this situation extremely seriously. We have immediately suspended this line of research."
If you've got a suggestion of a way to catch those bugs, please be more specific about it. Just telling people that they need "better protection" isn't really useful or actionable advice, or anything that they weren't already aware of.
Yes, it does.
Now, how do you do that other than having fallible people review things?
"Earth is center of universe" took 1000 years to remove from books, I'm not sure what her point was :D
However, the prior activity of submitting bad-faith code is indeed pretty shameful.
It's a different university, but I wonder if these people will see the same result.
Black list the whole lot from everything, everywhere. Black hole that place and nuke it from orbit.
Should every city park with a "no alcohol" policy conduct red teams on whether it's possible to smuggle alcohol in? Should police departments conduct red teams to see if people can get away with speeding?
I don't think they're a professor are they? Says they're a PhD student?
> Because of this, I will now have to ban all future contributions from your University.
Understandable from gkh, but I feel sorry for any unrelated research happening at University of Minnesota.
EDIT: Searching through the source code[1] reveals contributions to the kernel from umn.edu emails in the form of an AppleTalk driver and support for the kernel on PowerPC architectures.
In the commit traffic[2], I think all patches have come from people currently being advised by Kangjie Liu[3] or Liu himself dating back to Dec 2018. In 2018, Wenwen Wang was submitting patches; during this time he was a postdoc at UMN and co-authored a paper with Liu[4].
Prior to 2018, commits involving UMN folks appeared in 2014, 2013, and 2008. None of these people appear to be associated with Liu in any significant way.
[1]: https://github.com/torvalds/linux/search?q=%22umn.edu%22
[2]: https://github.com/torvalds/linux/search?q=%22umn.edu%22&typ...
- Aditya Pakki (the author who sent the new round of seemingly bogus patches) is not involved in the S&P 2021 research. This means Aditya is likely to have nothing to do with the prior round of patching attempts that led to the S&P 2021 paper.
- According to the authors' clarification [1], the S&P 2021 paper did not introduce any bugs into Linux kernel. The three attempts did not even become Git commits.
Greg has all reasons to be unhappy since they were unknowingly experimented on and used as lab rats. However, the round of patches that triggered his anger *are very likely* to have nothing to do with the three intentionally incorrect patch attempts leading to the paper. Many people on HN do not seem to know this.
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
What this professor is proving out is that open source and (likely, other) high trust networks cannot survive really mendacious participants, but perhaps by mistake, he's showing how important it is to make very harsh and public examples of said actors and their mendacity.
I wonder if some of these or other bug contributors have also complained that the culture of the project governance is too aggressive, that project leads can create an unsafe environment, and discourage people from contributing? If counter-intelligence prosecutors pull on this thread, I have no doubt it will lead to unravelling a much broader effort.
[1] https://cse.umn.edu/cs/statement-cse-linux-kernel-research-a...
This is overkill and uncalled for.
The foundation should use recourse to the law to signal they are handling it, if only to prevent these profs from being mobbed.
They claim that none of the Bogus patches were merged to the Stable code line :
>Once any maintainer of the community responds to the email,indicating “looks good”,we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
I haven't been able to find out what the 3 patches which the reference are, but the discussions on Greg's UMN Revert patch [2] does indicate that some of the fixes have indeed been merged to Stable and are actually Bogus.
[1] : https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
[2] : https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
[0] https://cse.umn.edu/cs/statement-cse-linux-kernel-research-a...
https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
See section VI-A (page 8).
>... learned today about the details of research being conducted by one of its faculty members and graduate students into the security of the Linux Kernel
And this sounds like mainly a lot of damage control is going to happen.
>We will report our findings back to the community as soon as practical.
Can someone who's more invested into kernel devel find them and analyze their impact? That sounds pretty interesting to me.
Edit: This is the patch reverting all commits from that mail domain: https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
Edit 2: Now that the first responses to the reversion are trickling in, some merged patched were indeed discovered to be malicious, like the following. Most of them seem to be fine though or at least non malicious. https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
Conducting such pen-tests, and then publishing the results openly, helps raise awareness about the need to assume-bad-faith in all OSS contributions. If some random grad student was able to successfully inject 4 vulnerabilities before finally getting caught, I shudder to think how many vulnerabilities were successfully injected, and hidden, by various nation-states. In order to better protect ourselves from cyberwarfare, we need to be far more vigilant in maintaining OSS.
Ideally, such research projects should gain prior approval from the project maintainers. But even though they didn't, this paper is still a net-positive contribution to society, by highlighting the need to take security more seriously when accepting OSS patches.
> A lot of these have already reached the stable trees.
If the researchers were trying to prove that it is possible to get malicious patches into the kernel, it seems like they succeeded -- at least for an (insignificant?) period of time.
[PATCH 000/190] Revertion of all of the umn.edu commitsOne thing I hope they have considered is a possible intent to get a particular important patch from umn.edu reverted to reintroduce a kernel bug. Discrediting all commits from the organization could inadvertently lead to the reintroduction of legacy exploits.
> This patchset has the "easy" reverts, there are 68 remaining ones that need to be manually reviewed. Some of them are not able to be reverted as they already have been reverted, or fixed up with follow-on patches as they were determined to be invalid. Proof that these submissions were almost universally wrong.
Skimming through subject lines of 190 commits being reverted here, every single one of them is along the lines of "add refcount/NULL/etc check and conditionally do (or do not) de-allocate memory before error-path return". I.e. worst case - this will reintroduce some rare memory leak or memory lifecycle bug.
Also, all of patches in question are in drivers. So depending on hardware used, any given system's user is likely to only have to worry about 2-3, maybe 5 of the patches, not all 190.
UMN looks pretty shoddy - the response from the researcher saying these were automated by a tool looks like a potential lie.
Or is this kind of experiment deemed fair game? Red vs blue team kind of thing? Penetration testing.
But if it was me in this situation, I'd ban them for ethics violation as well. Acting like a Evil doer means you might get caught... and punished. I found the email about cease and desist particularly bad behavior. If that student was lying then that university will have to take real action. Reputation damage and all that. Surely a academic reprimand.
I'm sure there's plenty of drama and context we don't know about.
Um. Ok.
That said the current incident seems to have gone beyond the limits of that one and is a new incident. I just thought it would be fair to include their "side"
With that said, kernel developers and companies with servers on the internet are busy doing work that's important to them. This sort of thing is always an unwelcome distraction.
And, if my neighbors walks in my door at 3 a.m. to let me know I left it unlocked, I'm going to treat them the same way UMN is getting treated in this situation. Or worse.
This was his clarification https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
...in which they have the nerve to say that this is not considered "human research". It most definitely is, given that their attack vector is the same one many people would be keen on using for submitting legitimate requests for getting involved.
If anything, this "research" highlights the notion that coding is but a small proportion of programming and delivery of a product, feature, or bugfix from start-to-finish is a much bigger job than many people like to admit to themselves or others.
> Attitude that is not only unwelcome but also intimidating to newbies and non experts
Between that and the "Clarifications" document suggesting they handle it by updating their Code of Conduct, they're clearly trying really hard to frame all of this as some kind of toxic culture in kernel development. That's a hideous defense. It's like a bad MMA fight where one fighter refuses to stand up because he insists on keeping it a ground fight. Maybe it works sometimes, but it's shameful.
You're just testing the review ability of particular Linux kernel maintainers at a particular point in time. How does that generalize to the extent needed for it to be valid research on open source software development in general?
You would need to run this "experiment" hundreds or thousands of times across most major open source projects.
I think it's mostly "finger pointing": you need one exception to break a rule. If the rule is "open source is more secure than closed source because community/auditing/etc.", now with a paper demonstrating that this rule is not always true you can write a nice Medium article for your closed-source product, quoting said paper, claiming that your closed-source product is more secure than the open competitor.
They were grossly wrong, of course. The work is extremely unethical. But I don't believe that their other actions are consistent with a "we hate OSS and want to prove it is bad" ethos.
Exactly how clever? That varies from reviewer to reviewer.
There will be large projects, with many people that review the code, which will not catch sufficiently clever vulnerabilities. There will be small projects with a single maintainer that will catch just about anything.
There is a spectrum. Without conducting a wide-scale (and unethical) survey with a carefully calibrated scale of cleverness for vulnerabilities, I don't see how this is useful research.
Unbelievable that this could have passed ethics review, so I'd bet it was never reviewed. Big black eye for University of Minnesota. Imagine if you are another doctoral student is CS/EE and this tool has ruined your ability to participate in Linux.
To mitigate the risks, we make several suggestions. First, OSS projects would be suggested to update the code of conduct by adding a code like "By submitting the patch, I agree to not intend to introduce bugs."
What's preventing those bad actors from not using a UMN email address?
Technically none, but by banning UMN submissions, the kernel team have sent an unambiguous message that their original behaviour is not cool. UMN's name has also been dragged through the mud, as it should be.
Prof Lu exercised poor judgement by getting people to submit malicious patches. To use further subterfuge knowing that you've been already been called out on it would be monumentally bad.
I don't know how far Greg has taken this issue up with the university, but I would expect that any reasonable university would give Lu a strong talking-to.
They gain some trust comming from university email addresses
If they just want to be jerks, yes. But they can't then use that type of "hiding" to get away with claiming it was done for a University research project as that's even more unethical than what they are doing now.
New plan: Show up at Liu's house with a lock picking kit while he's away at work, pick the front door and open it, but don't enter. Send him a photo, "hey, just testing, bro! Legitimate security research!"
That's the university's problem to fix.
Unless both the professors and leadership from the IRB aren't having an uncomfortable lecture in the chancellor's office then nothing at all changes.
There are a lot of people to feel bad for, but none is at the University of Minnesota. Think of the Austrians.
It's not wrong for the kernel community to decide to blanket ban contributions from the university. It obviously makes sense to ban contributions from institutions which are known to send intentionally buggy commits disguised as fixes. That doesn't mean you can't feel bad for the innocent students and professors.
This analogy is invalid, because:
1. The experiment is not on live, deployed, versions of the kernel.
2. There are mechanisms in place for preventing actual merging of the faulty patches.
3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
All of the above is not true for the in-flight airline.
However - I'm not claiming the experiment was not ethically faulty. Certainly, the U Minnesota IRB needs to issue a report and an explanation on its involvement in this matter.
The main problem is that they have (so far) refused to explain in detail how the patches where reviewed and how. I have not gotten any links to any lkml post even after Kangjie Lu personally emailed me to address any concerns.
I like that. That's what makes universities interesting to me.
I don't like the standard here of of penalizing or lumping everyone there together, regardless of they contribute in the past, now, in the future or not.
For example: what happens when the students graduate- does the ban follow them to any potential employers? Or if the professor leaves for another university to continue this research?
Does the ban stay with UMN, even after everyone involved left? Or does it follow the researcher(s) to a new university, even if the new employer had no responsibility for them?
if university has a problem, then they should first look into managing this issue at their end, or force people to use personal email ids for such purposes
If you don't manage to reach that goal, too bad, but you can contribute on a personal capacity, and/or go work elsewhere.
It sounds like you're making impossible demands of unrelated people, while doing nothing to solve the actual problem because the perpetrators now know to just create throwaway emails when submitting patches.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
(All of this ASSUMING that the intent was as described in the thread.)
They are conducting research to demonstrate that it is easy to introduce bugs in open source...
(whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards)
[removed this ranting that does not apply since they are contributing a lot to the kernel in good ways too]
> They are conducting research to demonstrate that it is easy to introduce bugs in open source...
That's a very dangerous thought pattern. "They try to find flaws in a thing I find precious, therefore they must hate that thing." No, they may just as well be trying to identify flaws to make them visible and therefore easier to fix. Sunlight being the best disinfectant, and all that.
(Conversely, people trying to destroy open source would not publicly identify themselves as researchers and reveal what they're doing.)
> whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards
How do we know that? We know things by regularly testing them. That's literally what this research is - checking how likely it is that intentional vulnerabilities are caught during review process.
This is a ridiculous conclusion. I do agree with the kernel maintainers here, but there is no way to conclude that the researchers in question "hate open source", and certainly not that such an attitude is shared by the university at large.
That's not true at all. There are many internet-critical projects with tons of holes that are not found for decades, because nobody except the core team ever looks at the code. You have to actually write tests, do fuzzing, static/memory analysis, etc to find bugs/security holes. Most open source projects don't even have tests.
Assuming people are always looking for bugs in FOSS projects is like assuming people are always looking for code violations in skyscrapers, just because a lot of people walk around them.
Which is why there have never been multi-year critical security vulnerabilities in FOSS software.... right?
Sarcasm aside, because of how FOSS software is packaged on Linux we've seen critical security bugs introduced by package maintainers into software that didn't have them!
[1] https://adityapakki.github.io/assets/files/aditya_cv.pdf
https://adityapakki.github.io/
In this "About" page:
https://adityapakki.github.io/about/
he claims "Hi there! My name is Aditya and I’m a second year Ph.D student in Computer Science & Engineering at the University of Minnesota. My research interests are in the areas of computer security, operating systems, and machine learning. I’m fortunate to be advised by Prof. Kangjie Lu."
so he in no uncertain terms is claiming that he is being advised in his research by Kangjie Lu. So it's incorrect to say his patches have nothing to do with the paper.
This being the internet, I'm sure the guy is getting plenty of hate mail as it is. No need to make it worse.
Professors usually work on multiple projects, which involve different grad students, at the same time. Aditya Pakki could be working on a different project with Kangjie Lu, and not be involved with the problematic paper.
I used to work as an auditor. We were expected to conduct our audits to neither expect nor not expect instances of impropriety to exist. However, once we had grounds to suspect malfeasance, we were "on alert", and conduct tests accordingly.
This is a good principle that could be applied here. We could bat backwards and forwards about whether the other submissions were bogus, but the presumption must now be one of guilt rather than innocence.
Personally, I would have been furious and said, in no uncertain terms, that the university keep a low profile and STFU lest I be sufficiently provoked to taking actions that lead to someone's balls being handed to me on a plate.
I'm no lawyer, but it seems like there'd be something actionable.
On a side note, this brings into question any research written by any of the participating authors, ever. No more presumption of good faith.
Except that at least one of those three, did [0]. The author is incorrect that none of their attempts became git commits. Whatever process that they used to "check different versions of Linux and further confirmed that none of the incorrect patches was adopted" was insufficient.
That doesn't appear to be one of the three patches from the "hypocrite commits" paper, which were reportedly submitted from pseudononymous gmail addresses. There are hundreds of other patches from UMN, many from Pakki[0], and some of those did contain bugs or were invalid[1], but there's currently no hard evidence that Pakki was deliberately making bad-faith commits--just the association of his advisor being one of the authors of the "hypocrite" paper.
[0] https://github.com/torvalds/linux/commits?author=pakki001@um...
[1] Including his most recent that was successfully applied: https://lore.kernel.org/lkml/YH4Aa1zFAWkITsNK@zeniv-ca.linux...
Unfair? Maybe: complain to your advisor.
Like you can go to any government building with a threat of bombs but claiming it is only an experiment to find security loophole.
In short "f** around, find out"
"We sought to probe vulnerabilities of the open-source public-development process, and our results include a methodology for getting an entire university's email domain banned from contributing."
From the post:
* Does this project waste certain efforts of maintainers?
Unfortunately, yes. We would like to sincerely apologize to the maintainers involved in the corresponding patch review process; this work indeed wasted their precious time. We had carefully considered this issue, but could not figure out a better solution in this study. However, to minimize the wasted time, (1) we made the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we tried hard to find three real bugs, and the patches ultimately contributed to fixing them
As someone not part of academia, how could this research be judged to not involve people? It _seems_ obvious to me that the entire premise is based around tricking/deceiving the kernel maintainers.
No one says "wasted their precious time" in a sincere apology. The word 'precious' here is exclusively used for sarcasm in the context of an apology, as it does not represent a specific technical term such as might appear in a gemology apology.
There IS a better solution: not to proceed with that "study" at all.
That is the perfect example of being arrogant
Couldn't figure out that "not doing it" was an option apparently.
I'm going to go with "both" here.
Not sure how the researchers didn't see how this would backfire, but it's a hopeless misuse of their time. I feel really bad for the developers who now have to spend their time fixing shit that shouldn't even be there, just because someone wanted to write a paper and their peers didn't see any problems either. How broken is academia really?
The researchers should have approached the maintainers got get buy in, and setup a methodology where a maintainer would not interfere until a code merge was immanent, and just play referee in the mean time.
That's because those are two separate incidents. The study which resulted in 3 patches was completed some time last year, but this new round of patches is something else.
It's not clear whether the patches are coming from the same professor/group, but it seems like the author of these bogus patches is a Phd student working with the professor who conducted that study last year. So there is at least one connection.
EDIT: also, those 3 patches were supposedly submitted using a fake email address according to the "clarification" document released after the paper was published. So they probably didn't use a @umn.edu email at all.
Now a different researcher from UMN, Aditya Pakki, has submitted a patch which contains bugs that seems to be attempting to do the same type of pen testing although the PhD student denied it.
1. Section IV.A of the paper, as pointed out by user MzxgckZtNqX5i in this comment: https://news.ycombinator.com/item?id=26890872
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
2. Clarifications on the “hypocrite commit” work (FAQ)
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
"* Does this project waste certain efforts of maintainers? Unfortunately, yes. We would like to sincerely apologize to the maintainers involved in the corresponding patch review process; this work indeed wasted their precious time. We had carefully considered this issue, but could not figure out a better solution in this study. However, to minimize the wasted time, (1) we made the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we tried hard to find three real bugs, and the patches ultimately contributed to fixing them."
With more than 60% of all acedemic publications not being reproducible [1], one would think academia has better things to do than wasting other people's time.
Without trust, these projects will fail. Research has shown that even in the presence of untrustworthy actors, trusting is usually still beneficial [1][2]. Instead, trust until you have reason to believe you shouldn't has been found to be an optimal strategy [2], so G K-H is responding exactly appropriately here. The linux community trusted them until they didn't, and now they are unlikely to trust them going forward.
[1] https://www.nature.com/articles/s41598-019-55384-4#Sec13 [2] https://medium.com/greater-than-experience-design/game-theor...
Would I prefer to live in a world where everyone behaved in a trustworthy manner in OSS? Absolutely. But that is not the world we live in. A professor highlighting this fact, and forcing people to realize the dangers in trusting people, does more good than harm.
--------------
On a non-serious and humorous note, this episode reminds me of the Sokal Hoax. Most techies/scientists I've met were very appreciative of this hoax, even though it wasn't conducted with pre-approval from the subjects. It is interesting to see the shoe on the other foot
There's claims that one vulnerability got committed and was not reverted by the research group. In fact the research group didn't even notice that it got committed. So I'd argue that this was a net negative to society because it introduced a live security vulnerability into linux.
There are a lot of people reading these discussions who aren't taking 'sides' but trying to think about the subject. Looking at different angles helps with thinking.
A software as critical as linux should not be this easily compromised by a bunch of grads..
It's one of the core technologies of our computing.
Having a discussion around the ethics of this is great but it does not detract from the importance of bigger issue.
While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best.
Two messages down in the same thread, it more or less culminates with the university e-mail suffix being banned from several kernel mailing lists and associated patches being removed[1], which might be an appropriate response to discourage others from similar stunts "for science".
[1] https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
Ensuring the safety of the experiment. In the experiment, we aim to demonstrate the practicality of stealthily introducing vulnerabilities through hypocrite commits. Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF bugs will not be merged into the actual Linux code. In addition to the minor patches that introduce UAF conditions, we also prepare the correct patches for fixing the minor issues. We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch.
Are you saying that despite this, these malicious commits made it to production?
Taking the authors at their word, it seems like the biggest ethical consideration here is that of potentially wasting the time of commit reviewers—which isn't nothing by any stretch, but is a far cry from introducing bugs in production.
Are the authors lying?
I agree. I would say this is kind of a "human process" analog of your typical computer security research, and that this behavior is akin to black hats exploiting a vulnerability. Totally not OK as research, and totally reckless!
Haven't whitehat hackers doing unsolicited pen-testing been prosecuted in the past?
From FOSDEM 2014, NSA operation ORCHESTRA annual status report. It’s pretty entertaining and illustrates that this is nothing new.
https://archive.fosdem.org/2014/schedule/event/nsa_operation... https://www.youtube.com/watch?v=3jQoAYRKqhg
Very good point.
In a roundabout way, this researcher has achieved their goal, and I hope they publish their results. Certainly more meaningful than most of the drivel in the academic paper mill.
This will lead in effect to that even valuable contributions from universities will be seen with more suspicion and will be very damaging in the long run.
What review process catches 100% garbage? It's a mechanism to catch 99% of garbage -- otherwise Linux kernel would have no bugs.
Does that add anything new to what we know since the creation of the "obfuscated C contest" in 1984?
It shows nothing of the sort. No review process is 100% foolproof, and opensource means that everything can be audited if it is important to you.
The other option is closed source everything and I can guarentee that review processes let stuff through, even if its only "to meet deadlines" and you will unlikely be able to audit it.
did these "researchers" in any way demonstrate that they were going to come clean about what they had done before their "research" made to anywhere close to release/GA?
Try to introduce yourself in the White House and when you get caught tell them "I was just testing your security procedures".
"if they are attempting to test us by first submitting malicious patches as an experiment, we can't accept what we have accepted as not being malicious and so it's safer to remove them than to keep them".
my 2c.
Obviously, trust should not be the only thing that maintainers rely on, but it is a social endeavour and trust always matters in such endeavors. Doing business with people you can't trust makes no sense. Without trust I agree fully that it is not worth the maintainer's time to accept anything from such people, or from that university.
And the fact that one can do damage with malicious code is nothing new at all. It is well known and nothing new that bad code can ultimately kill people. It is also more than obvious that I can ring the door of my neighbor, ask him or her for a cup of sugar, and blow a hammer over their head. Or people can go to a school and shoot children. Does anyone in his right mind has to do such damage in order to prove something? No. Does it prove anything? No. Does the fact that some people do things like that "prove" that society is wrong and trust and collaboration is wrong? What an idiocy, of course not!
Before this study came out, I'm pretty sure there were already known examples of this happening, and it would have been reasonable to assume that some such vulnerabilities existed. But now we have even more reason to worry, given that they succeeded doing this multiple times as a two person team without real institutional backing. Imagine what a state-level actor could do.
Each institution, and each IRB is made up of people and a set of policies. One does not have to meaningfully misrepresent things to IRBs for them to be misunderstood. Further, exempt from IRB review and 'not human subjects research' are not actually the same thing. I've run into this problem personally - IRB declines to review the research plan because it does not meet their definition of human subjects research, however the journal will not accept the article without IRB review. Catch-22.
Further, research that involves deception is also considered a perfectly valid form of research in certain fields (e.g., Psychology). The IRB may not have responded simply because they see the complaint as invalid. Their mandate is protecting human beings from harm, not random individuals who email them from annoyance. They don't have in their framework protecting the linux kernel from harm any more than they have protecting a jet engine from harm (Sorry if that sounds callous). Someone not liking a study is not research misconduct and if the IRB determined within their processes that it isn't even human subjects research, there isn't a lot they can do here.
I suspect that this is just one of those disconnects that happens when people talk across disciplines. no misrepresentation was needed, all that was needed was for someone reviewing this, who's background is medicine and not CS, to not understand the organizational and human processes behind submitting a software 'patch'.
The follow up behavior...not great...but the start of this could be a serious of individually rational actions that combine into something problematic because they were not holistically evaluated in context.
They also lied in the paper about their methodology - claiming that once their code was accepted, they told the maintainers it should not be included. In reality, several of their bad commits made it into the stable branch.
To be clear, this is unethical research.
But I read the paper, and these patches were probably automatically generated by a tool (or perhaps guided by a tool, and filled in concretely by a human): their analyses boil down to a very simple LLVM pass that just checks for pointer dereferences and inserts calls to functions that are identified as performing frees/deallocations before those dereferences. Page 9 and onwards of the paper[1] explains it in reasonable detail.
[1]: https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
Could they have submitted patches to fix the problems based on same tooling or was that not possible (I am not close to kernel development flow)?
(3). We send the incorrect minor patches to the Linux community through email to seek their feedback.
(4). Once any maintainer of the community responds to the email, indicating “looks good”, we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
------------------------
But this shows a distinct lack of understanding of the problem:
> This is not ok, it is wasting our time, and we will have to report this,
> AGAIN, to your university...
------------------------
You do not experiment on people without their consent. This is in fact the very FIRST point of the Nuremberg code:
1. The voluntary consent of the human subject is absolutely essential.
Exactly this. Research involving human participants is supposed to have been approved by the University's Institutional Review Board; the kernel developers can complain to it: https://research.umn.edu/units/irb/about-us/contact-us
It would be interesting to see what these researches told the IRB they were doing (if they bothered).
Edited to add: From the link in GP: "The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained)"
Okay so this IRB needs to be educated about this. Probably someone in the kernel team should draft an open letter to them and get everyone to sign it (rather than everyone spamming the IRB contact form)
T
If this behaviour is tolerated by the University of Minnesota (and it appears to be so) then I suppose that's another institution on my list of unreliable research.
I do wonder what the legal consequences are. Would knowingly and willfully introducing bad code constitute a form of vandalism?
Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?
From a common sense standpoint, it seems to me this is more about medical experiments. Yesterday I put some of my kids toys away without telling them to see if they’d notice and still play with them. I don’t think I need IRB approval.
I think it shows that this type of study might well be needed, it just needs to be done better and with the consent of the maintainers.
As I understand it, any "experiment" involving other people that weren't explicitly informed of the experiment before hand needs to be a lot more carefully considered than what they did here.
> I respectfully ask you to cease and desist from making wild accusations that are bordering on slander.
> These patches were sent as part of a new static analyzer that I wrote and it's sensitivity is obviously not great. I sent patches on the hopes to get feedback. We are not experts in the linux kernel and repeatedly making these statements is disgusting to hear.
( https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah... )
How does that fit in with your explanation?
> 1. The voluntary consent of the human subject is absolutely essential.
The Nuremberg code is explicitly about medical research, so it doesn't apply here. More generally, I think that the magnitude of the intervention is also relevant, and that an absolutist demand for informed consent in all - including the most trivial - cases is quite silly.
Now, in this specific case I would agree that wasting people's time is an intervention that's big enough to warrant some scrutiny, but the black-and-white way of some people to phrase this really irks me.
PS: I think people in these kinds of debate tend to talk past one another, so let me try to illustrate where I'm coming from with an experiment I came across recently:
To study how the amount of tips waiters get changes in various circumstances, some psychologists conducted an experiment where the waiter would randomly either give the guests some chocolate with the bill, or not (control condition)[0] This is, of course, perfectly innocuous, but an absolutist claim about research ethics ("You do not experiment on people without their consent.") would make research like this impossible without any benefit.
[0] https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1559-1816...
For that matter, what's the difference between this and pentesting?
I wonder how many zero days have been included already, for example by nation state actors...
If I were at the receiving end, I’d think checking a patch multiple times before accepting it.
>1. The voluntary consent of the human subject is absolutely essential.
Does this also apply to scrapping people's data?
By this logic eg. resume callback studies aiming to study bias in the workforce would be impossible.
> 1. The voluntary consent of the human subject is absolutely essential.
Which is rather useless, as for many experiments to work, participants have to either be lied to, or kept in the dark as to the nature of the experiment, so whatever “consent” they give is not informed consent. They simply consent to “participate in an experiment” without being informed as to the qualities thereof so that they truly know what they are signing up for.
Of course, it's quite common in the U.S.A. to perform practice medical checkups on patients who are going under narcosis for an unrelated operations, and they never consented to that, but the hospitals and physicians that partake in that are not sanctioned as it's “tradition”.
Know well that so-called “human rights” have always been, and shall always be, a show of air that lack substance.
Like somebody picking your locks, and suggesting, 'to stop this one approach would be to post a sign "do not pick"'
from https://www-users.cs.umn.edu/~kjlu/
If the original research results in a paper and IEEE conference presentation, why not? There's no professional consequences for this conduct, apparently.
These ~200 patches from UMN being reverted might have nothing to do with these researchers at all.
Hopefully someone from the university clarifies what's happening soon before the angry mob tries to eat the wrong people.
EDIT: Also, even if you do no harm and immediately inform your victim, this sort of stuff might rather be categorized as grey-hat. Maybe a "true" white-hat would only hack a system with explicit consent from the owner. These terms are fuzzy. But my point is, attacking a system for personal gain without notifying your victim afterwards and leaving behind malicious code is certainly not white-hat by any definition.
In order to do this ethically, all that's needed is respect towards our fellow human beings. This means informing them about the nature of the research, the benefits of the collected data, the risks involved for test subjects as well as asking for their consent and permission to be researched on. Once researchers demonstrate this respect, they're likely to find that a surprising number of people will allow them to perform their research.
We all hate it when big tech tracks our every move and draws all kinds of profitable conclusions based on that data at our expense. We hate it so much we deploy active countermeasures against it. It's fundamentally the same issue.
Not only that, but they are also doing experiments on a community of people which is against their interest and also could be harmful by creating mistrust. Trust is a big issue, without it it is almost impossible for people to work meaningfully together.
This has not been a valid excuse since the 1950s. Scientists are not allowed to ignore basic ethics because they want to discover something. Deliberately introducing bugs into any open source project is plainly unethical; doing so in the Linux kernel is borderline malicious.
The thing that people should be upset about is that such an important open source project so easily accepts patches which introduce security vulnerabilities. Forget the researchers for a moment - if it is this easy, you can be certain that malicious actors are also doing it. The only difference is that they are not then disclosing that they have done so!
The Linux maintainers should be grateful that researchers are doing this, and researchers should be doing it to every significant open source project.
didn't we learn a lot from nazi/japanese experiments from ww2?
Though what a lot is is also open to question. Much of what we learned isn't that useful to real world problems. However some has been important.
The goal of military is to protect or conquer. The goal of science is to find the truth, and the goal of the engineering is to offer solutions. Any of the true leaders in either fields knows there're more efficient means/systems to get those goals, even in ww2 era.
And ultimately, we know what their priorities were and what kind of worldview they were operating under, so the odds are bad that any given experiment they ran would have been rigorous enough to produce results that could be reproduced in other studies and applied elsewhere. I'm not personally aware of any major breakthroughs that would have been impossible without the "aid" of eugenicist war criminals, though it's possible there's some major example I'm missing.
We certainly did bring over lots of German scientists to work on nukes and rockets, so your question is not entirely off-base - but I suspect almost everyone involved in those choices would argue that rocketry research isn't unethical.
> But they can't then use that type of "hiding" to get away with claiming it was done for a University research project as that's even more unethical than what they are doing now.
Other CS labs at UMN, well... apparently not so much.
A good example is challenge testing Covid vaccines. This was widely deemed to be unethical despite large numbers of volunteers. Perhaps a million lives could have been saved if we had vaccines a few months sooner.
Research without ethics (as currently practiced) can have value.
Issue (2) arose with the EU response to rare AZ/J+J side effects, where I believe the EU is more deserving of criticism. They will undoubtedly cause more deaths in their own populations and throughout the world than would occur from clotting complications, but no one will hold them to account. But they weighed their equities as more important than global benefit.
This does not at all mean the behavior in question should be condoned. This fails the sniff test worse than thioacetone.
In general, it is the wrong attitude to say, oh we had a security problem. What a fiasco! Everyone involved should be fired! With a culture like that, all you guarantee is that people cover up the security issues that inevitably occur.
Perhaps this incident actually does indicate that kernel code review procedures should be changed in some way. I don’t know, I’m not a kernel expert. But the right way to do that is with a calm postmortem after appropriate immediate actions are taken. Rolling back changes made by malicious actors is a very reasonable immediate action to take. After emotions have cooled, then it’s the right time to figure out if any processes should be changed in the future. And kernel devs putting in extra work to handle security incidents should be appreciated, not criticized for their imperfection.
Kind of like:
A: we're going to experiment with humans.
C: are you going to extract fluids?
A: no.
C: (ticks no) are you going to cut into them?
A: no.
C: (ticks no) ...
And so on.
Perhaps a new set of questions might help.
C: is what you are intending going to anger the subjects to the point they will take retribution?
A: yes
C: (ticks yes) how will it anger them?
....
And then expand from there. I'm sure they don't just stay within the checklist like robots.
Then again I'm probably wrong. Its just my imagination. But it could be true.
As for systematic issues, I'm not sure. But moving forward they'd want to confirm there aren't glaring omissions to let this happen again. Giving them suitable Benefit-of-doubt niceties might imply these are isolated cases. (But both of them?! Perhaps isolated to a small group of academics.)
Messy situation.
...which is why the interestingness of this project depends on how clever they were - which I'm not able to evaluate, but which someone would need to before they could possibly invalidate the idea.
> (and unethical)
How is security research unethical, exactly?
Those being researched must consent.
The goal should be to further society. This research attempted to sabotage infrastructure.
Research should avoid unnecessary suffering. Kernel maintainers are overworked volunteers.
They must be allowed to discontinue the research if the stress becomes more than they can bear.
Read more on University of Minnesota's website and look at page 4. https://www.ahc.umn.edu/img/assets/26104/Research_Ethics.pdf
In the case of research, universities are required to have an ethics board that reviews research proposals before actual research is conducted. Conducting research without an approval or misrepresenting the research project to the ethics board are pretty serious offenses.
Typically for research that involves people, participants in the research require having a consent form that is signed by participants, alongside a reminder for participants that they can withdraw that consent at any time without any penalties. It's pretty interesting that in this case, there seemed to have been no real consent required, and it would be interesting to know whether there has been an oversight by the ethics board or a misrepresentation of the research by the researchers.
It will be interesting to see whether the university applies a penalty to the professor (removal of tenure, termination, suspension, etc.) or not. The latter would imply that they're okay with unethical or misrepresented research being associated with their university, which would be pretty surprising.
In any case, it's a good thing that the Linux kernel maintainers decided that experimenting on them isn't acceptable and disrespectful of their contributions. Subjecting participants to experiments without their consent is a severe breach of ethical duty, and I hope that the university will apply the correct sanctions to the researchers and instigators.
Of course, in a few years this will all be forgotten. It begs the question... how effective is it to ban entire organizations due to the actions of a few people? Part of me thinks that it would very good to have something like this happen every five years (because it puts the maintainers on guard), but another part of me recognizes that these maintainers are working for free, and they didn't sign up to be gaslighted, they signed up to make the world a better place. It's not an easy problem.
I don't think it's unreasonable for maintainers of software to ignore or outright ban problematic users/contributors. It's up to them to manage their software project the way they want, and if banning organizations with malicious actors is the way to do it, the more power to them.
[1] https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
it was, iirc, a poster not a paper.
"Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction" (2010, in Journal of Serendipitous and Unexpected Results)
We had the good fortune to have discussion of the study with comments from the author a few years back:
Intentionally having bugs in kernel only you know about is very bad.
You can't break into someone's house through their back window, tell the owners what you did, and not expect to get arrested.
People don't scream "how are we going to know that people can break into houses through broken windows without these heros!?"
Really losing my faith in the accuracy of HN if such a huge thread is full of misinformation.
Basically (as I understand it, feel free to correct me) this is what happened:
Researcher emailed maintained with flawed code, maintainer LGTMed it, researcher told maintainer that the code is buggy and not to merge it. The researchers confirmed that the code was not merged or commited anywhere. Paper gets published. Nothing of note happens.
Now, one of the researchers grad students has submitted stuff to linux oh his own volition- he does not appear to be associated with the previous research. These commits are "obviously bad" according to linux maintainers and claim that the grad student is just continuing the "merge bad shit" research. These commits do not appear to be intentionally flawed but rather newbie mistakes (so claims the student)- which is why he feels the linux community is unwelcoming to newcomers.
Now how on earth did that warp to whatever everyone here is smoking?
I know the kernel doesn't need anyone's contributions anyhow, but as a matter of policy this seems like a bad one.
Low trust and negative trust should be fairly obvious costs to messing with a trust model - you could easily argue this is working as intended.
It stays with the university until the university provides a good reason to believe they should not be particularly untrusted.
It's a chain that gets really unpleasant.
I am also not a lawyer, but aside from any civil action, the conduct looks like it might be considered criminal under the Computer Fraud and Abuse Act:
"Whoever knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer;"
The first extraterrestrial software crime?
[0] https://www.theverge.com/2021/2/19/22291324/linux-perseveran...
https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
https://lore.kernel.org/linux-nfs/CADVatmNgU7t-Co84tSS6VW=3N...
I trust Greg over the students.
"""Romanovsky reported that he had looked at four accepted patches from Pakki "and 3 of them added various severity security 'holes.'" Sudip Mukherjee, Linux kernel driver and Debian developer, followed up and said "a lot of these have already reached the stable trees." These patches are now being removed."""
However, if you click the links, you'll see that "have already reached stable trees" is about non-buggy patches, and "3 of them added various [holes]" are not one of those. So the articles seem to be intentionally deceiving the reader to think those are connected, when they're separate events. I actually feel like the media has been doing this (putting together non-related facts together in a way that readers reasonably infer a connection between the two).
Russia and China effectively approved vaccines with only phase II trial data. China didn't even have active cases (officially) and they vaccinated tens of millions of people with it, their own, Africans and Brazilians etc. Imagine if there had been side effects, it could have been a global disaster. (Recall the contaminated Cutter polio vax that had live polio in it. Or the SV40 problems. The UQ COVID candidate caused false positives for HIV. Various SARS-Cov1 vaccines caused bad/lethal side effects, see e.g. https://www.nature.com/articles/s41579-020-00462-y )
Being both too hasty (China) and too blame-averse (EU) seem to be ethical failings.
Also, IRB review is only for research funded by the federal government. If you’re testing your kid’s math abilities, you’re doing an experiment on humans, and you’re entirely responsible for determining whether this is ethical or not, and without the aid of an IRB as a second opinion.
Even then, successfully getting through the IRB process doesn’t guarantee that your study is ethical, only that it isn’t egregiously unethical. I suspect that if this researcher got IRB approval, then the IRB didn’t realize that these patches could end up in a released kernel. This would adversely affect the users of billions of Linux machines world–wide. Wasting half an hour of a reviewer’s time is not a concern by comparison.
Usually when an organization is pen-tested it consented to being pen-tested (likely even requesting it).
Here there were no contact with the Linux foundation to gain consent for the experiment.
Right. It's not just human subjects research. IRBs vet all kinds of research: polling, surveys, animal subjects research, genetics/embryo research (potentially even if not human/mammal), anything which could be remotely interpreted as ethically marginal.
It's a real shame because the university probably has good, experienced people who could contribute to various OSS projects. But how can you trust any of them when the next guy might also be running an IRB exempt security study.
I don't think code commits to the Linux kernel make it to live systems that fast?
I do agree with the sentiment, though. It's grossly irresponsible to do that without asking at least someone in the kernel developer's group. People don't dig being used as lab rats, and now the whole uni is blocked. Well, tough shit.
edit: Reference to combat the downvote: https://www.nbcnews.com/news/china/american-universities-are...
A maintainer pakage is just one more open source software (thus also in need of reviews and audits)... which is why some people prefer upstream-source-based distribs, such as Gentoo, Arch when you use git-based AUR packages, or LFS for the hardcore fans.
Yes, you do need to make that comparison. Taking it as a given without analysis is the same as trusting the proprietary software vendors who claim to have robust QA on everything.
Security is hard work and different from normal review. The number of people who hypothetically could do it is much greater than the number who actually do, especially if there isn’t an active effort to support that type of analysis.
I’m not a huge fan of this professor’s research tactic but I would ask what the odds are that, say, an intelligence agency isn’t doing the same thing but with better concealment. Thinking about how to catch that without shutting down open-source contributions seems like an important problem.
If the patches were accepted, the person could have used those fixes to justify the benefits of the static analysis tool they wrote.
For instance:
"D. Feedback of the Linux Community. We summarized our findings and suggestions, and reported them to the Linux community. Here we briefly present their feedback. First, the Linux community mentioned that they will not accept preventive patches and will fix code only when it goes wrong. They hope kernel hardening features like KASLR can mitigate impacts from unfixed vulnerabilities. Second, they believed that the great Linux community is built upon trust. That is, they aim to treat everyone equally and would not assume that some contributors might be malicious. Third, they mentioned that bug-introducing patches is a known problem in the community. They also admitted that the patch review is largely manual and may make mistakes. However, they would do their best to review the patches. Forth, they stated that Linux and many companies are continually running bug-finding tools to prevent security bugs from hurting the world. Last, they mentioned that raising the awareness of the risks would be hard because the community is too large."
[1] https://raw.githubusercontent.com/QiushiWu/qiushiwu.github.i...
I doubt HN has the volume of readership/temperament to lead to substantial hate mail (unlike, say, Twitter).
You seem convinced, I am not. And as such, we just have a disagreement, no biggie. :)
>This sort of linguistic weaseling doesn't help anyone, least of all folks who may not understand the entire context of what went down here (which is much worse than it might seem on the face of it).
I am expressing an opinion, just as you are.
Maybe that cop convicted yesterday was actually just a UMN researcher investigating the burning scientific question "does cutting off someone's airway for 9 minutes cause death?".
Depends on what you mean: they knew exactly what they were patching, so they could easily have submitted inverse patches. On the other hand, the obverse research problem (patching existing UAFs rather than inserting new ones) is currently unsolved in the general case.
And further, pretty much everybody knows that malicious actors - if they tried hard enough - would be able to sneak through hard to find vulns.
And this is anything new?
And if I blow a hammer over your head while you are not suspecting it, does this prove anything else than that I am thug? Does it help you? Honestly?
Runs counter to how open source is ideally written, but for such a core project, perhaps stronger checks are needed.
Fascinating. Can you provide links?
https://ctexaminer.com/2021/03/20/explicit-consent-for-pelvi...
https://www.forbes.com/sites/paulhsieh/2018/05/14/pelvic-exa...
Most one can find of it also only deals with “intimate parts”; I am quite sceptical that this is the only thing that medical students require practice on and I think it more likely that the media only cares in this case and that in fact it is routine with many more body parts.
This isn't a "gotcha" - people shouldn't do this.
Here's another idea: If it's ethical to do it in a non-experimental context, it's also ethical to do it in an experimental context. So if it's OK to walk up to a stranger and ask them a weird question, it's also OK to do it in the context of a Youtube social experiment. Anything other than this is blatantly anti-scientific IMO.
It is IRBs that need reform. They're self-justifying bureaucratic cruft: https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
I've heard it many times that they're thoroughly reviewed and back doors are very unlikely. So yes, some people were under the impression.
And one of the most reasonable "stories" behind a patch is "I'm working at a university and found a bug", probably right behind "I'm working at a company and we found a bug".
Banning U of M won't solve everything, but it is dropping a source of known bad patches.
Wait so do you disagree with ZDnet too?
Did you read the ZDnet article and look at the links that in that article in the relevant paragraph? I'm not "disagreeing", I'm saying that they are misleading the reader (and it looks like many were fooled).
The two sentences they put together are not related, but put next to each other, they make it seem like they're related. We have to be careful when reading these articles. So the researchers have made commits to stable, and the researchers have introduced vulnerabilities, but they are not referring to the same patches. So no vulnerabilities have been committed to stable.
The damage is not that big. Only 4 committers to linux in the last decade, 2 of them, the students, with malicious backdoors, the Prof not with bad code but bad ethics, and the 4th, the Ass Prof did good patches and already left them.
I worked on a number of studies through undergrad and grad school, mostly involving having people test software. The work to get a study approved was easily 20 hours for a simple "we want to see how well people perform tasks in the custom software we developed. They'll come to the university and use our computer to avoid security concerns about software security bugs". You needed a script of everything you would say, every question you would ask, how the data would be collected, analyzed, and stored securely. Data retention and destruction policies had to be noted. The key linking a person's name and their participant ID had to be stored separately. How would you recruit participants, the exact poster or email you intend to send out. The reading level of the instructions and the aptitude of audience were considered (so academic mumbo jumbo didn't confuse participants).
If you check the box that you'll be deceiving participants, there was another entire section to fill out detailing how they'd be deceived, why it was needed for the study, etc. Because of past unethical experiments in the academic world, there is a lot of scrutiny and you typically have to reveal the deception in a debriefing after the completion of the study.
Once a study was accepted (in practice, a multiple month process), you could make modifications with an order of magnitude less effort. Adding questions that don't involve personal information of the participant is a quick form and an approval some number of days later.
If you remotely thought you'd need IRB approval, you started a conversation with the office and filled out some preliminary paperwork. If it didn't require approval, you'd get documentation stating such. This protects the participants, university, and professor from issues.
--
They took it really seriously. I'm familiar with one study where participants would operate a robot outside. An IRB committee member asked what would happen if a bee stung the participant? If I remember right, the resolution was an epipen and someone trained in how to use it had to be present during the session.
--
[0]: https://www.statista.com/statistics/272307/market-share-fore...
I'll just leave my comment as it is. The university administration still bears responsibility in the fact that they waived the IRB.
> Those commits are part of the following research:
> https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
> They introduce kernel bugs on purpose. Yesterday, I took a look on 4 accepted patches from Aditya and 3 of them added various severity security "holes".
If you mean supervisors adding their names on to publications without having contributed any work, than this is not only limited to CS research groups. Authorship misrepresentation is widespread in academia and unfortunately mostly being ignored. Those who speak up are being singled out and isolated instead.
It was written while I was working on the software and my name was then put in third position on the list of authors. Only way I was able to defend myself was, asking them to remove my name from the paper which resulted in the paper not being published.
> We send the emails to the Linux communityand seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter. The experiment will not collect any personal data, individual behaviors, or personal opinions. It is limited to studying the patching process OSS communities follow, instead of individuals.
So they did think of that. Either they misconstrued their research or the IRB messed up. Either way, they can now see for themselves exactly how human a pissed off maintainer is.
They did not say that they were hoping for feedback on their tool when they submitted the patch, they lied about their code doing something it does not.
>How does that fit in with your explanation?
It fits in the narrative of doing hypocritical changes to the project.
edit: oh, ok I guess that post with the accusations was mid-performance? Not inconsistent, so, maybe (I'm still not clear what the timeline is).
They obviously were _NOT_ created by a static analysis tool that is of
any intelligence, as they all are the result of totally different
patterns, and all of which are obviously not even fixing anything at
all. So what am I supposed to think here, other than that you and your
group are continuing to experiment on the kernel community developers by
sending such nonsense patches?
When submitting patches created by a tool, everyone who does so submits
them with wording like "found by tool XXX, we are not sure if this is
correct or not, please advise." which is NOT what you did here at all.
You were not asking for help, you were claiming that these were
legitimate fixes, which you KNEW to be incorrect.Sounds like they knew exactly what they were doing.
are there more impactful benefits that aren't listed on the wikipedia page? It sounds like the main research contribution is philosophical discussions over whether or not it would be okay to use the data if someone had a good reason for it.
Nevertheless it is clear that UMN does not have sufficient controls in place to prevent this kind of unethical behavior. The ban & patch reversions may force the issue.
UMN CS associate department head on this: https://twitter.com/lorenterveen/status/1384965111014641667 (TL;DR: they didn't hear about this before because each group does its thing, leadership doesn't get involved in IRB process, in his opinion IRB failed - situation analogous to cases known to be problematic in other subfields https://twitter.com/lorenterveen/status/1384955467051454466 )
from Lu's list of publications at https://www-users.cs.umn.edu/~kjlu/
Seems like a conference presentation at IEEE at minimum?
> If a paper raises significant ethical and/or legal concerns, it might be rejected based on these concerns.
https://www.ieee-security.org/TC/SP2021/cfpapers.html
So if the kernel maintainers report the issue to the S&P PC, the paper could potentially be rejected.
They were trusting of contributors to not be malicious, and in particular, were trusting of a university to not be wholly malicious.
Sure, there is a possible threat model where they would need to be suspicious of entire universities.
But in general, human projects will operate under some level of basic trust, with some sort of means to establish that trust. To be able to actually get anything done; you cannot perfectly formally review everything with finite human resources. I don't see where they went wrong with any of that here.
There's also the very simple fact that responding to an incident is also a part of the security process, and broadly banning a group whole-cloth will be more secure than not. So both them and you are getting what you want it of it - more of the process to research, and more security.
If the changes didn't make it out to production systems, then it seems like the process worked? Even if some of it was due to admissions that would not happen with truly malicious actors, so too were the patches accepted because the actors were reasonably trusted.
As with all human projects, some level and balance of trust and security is needed to get work done. And the gradient shifts as downstream forks have higher security demands / less trust, and (in the case of nation states) more resources and time to both move slower, validate changes and establish and verify trust.
For comparison, imagine that you attented a small conference and unknowingly became a test subject, and when you sit down the chair shocks you with a jolt of electricity. You jump out of your chair and exclaim, "This seat shocked me!" Then the person giving the presentation walks to your seat and sits down and it doesn't shock him (because he's the one holding the button), and he then accuses you of wasting everyone's time. That's essentially what happened here.
Disgusting.
All you have to do is look at the reverted patches to see that these are either mythical or at least few and far in between.
It's justifiable and natural for our name to be dragged in the mud here, but as a run of the mill software engineer who graduated from UMN, I hope our reputation isn't soured too much.
So at least definitely not "totally mythical".
Personally, I think all contributors should be considered "bad actors" in open source software. NSA, some university mail address, etc. I consider myself a bad actor, whenever I write code with security in mind. This is why I use fuzzing and code analysis tools.
Banning them was probably the correct action, but not finding value requires intentionally ignoring the very real result of the exercise.
However I'd also like to note that in a real world penetration test on an unwitting and non-consensual company, you also get sent to jail.
Everybody wins! The team get valuable insight on the security of the current system and unethical researchers get punished!
You don't get to rob a bank and then when caught say "you should thank us for showing your security weaknesses".
In this case they merged actual bugs and now they have to revert that stuff which depending on how connected those commits are to other things could cost a lot of time.
If they were doing this in good faith, they could have stopped short of actually letting the PRs merge (even then it's rude to waste their time this way).
This just comes across to me as an unethical academic with no real valuable work to do.
Any patch coming from somebody having intentionally introduced an issue falls into this category.
So, banning their organization from contributing is exactly the lesson to be learned.
So you admit it was a malicious breach? Of course it isn't a perfect process. Everyone knows it isn't absolutely perfect. What kind of test is that?
I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.
The question should also be due to who's neglect they gained access to the "water supply". If you also truly want to make this comparison.
It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose.
Edit: NM, see st_goliath response below
I fixed my sentence.
I still think that these professors, either genuinely or by lack of willingness, do not understand the mechanism by which free software warrants its greater quality compared to proprietary ones (which is a fact).
They just remind me the good old days of FUD against open source by Microsoft and its minions...
[Edited: it seems like they do love OSS and contribute a lot. See child comments.]
I had based my consideration on the way they are testing the open-source development model.
These professors actually love OSS... but they need to respect kernel maintainers request to stop these "experiments".
It's pretty obvious what would happen if a firm tried this: they'd be taken to court and probably imprisoned, as this is a clear violation of the law (which is pretty broadly to capture any attempted interference with the correct operation of a computer program).
So it’s possible that this situation has nothing to do with that research, and is just another unethical thing that coincidentally comes from the same university. Or it really is a new study by the same people.
Either way, I think we should get the facts straight before the wrong people are attacked.
Is it known whether these commits were indeed bad? It is certainly worth removing them just in case, but is there any confirmation?
The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either).
As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing.
What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money.
For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith.
Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns.
But what technical measure would you propose that can effectively stop con men?
It is unlikely that a business conducting A/B testing or a parent interacting with their children are receiving federal funds to support it. Therefore, their work is not subject to IRB review.
Instead, if you are a researcher who is funded by federal funds (even if you are doing work on your own children), you have to receive IRB approval for any work involving human interaction before you start conducting it.
Potentially yes, actually.
I still think it should be possible to run some A/B tests, but a lot depends on the underlying motivation. The distance between such tests and malicious psychological manipulation can be very, very small.
Psychology and sociology are both subject to the IRB as well.
Regardless of their department, this feels like a psychology experiment.
I would argue that ordinary A/B tests, by their very nature, are not "experiments" in the sense that restriction is intended for, so there is no reason for them to be considered unethical.
The difference between an A/B test and an actual experiment that should require the subjects' consent is that either of the test conditions, A or B, could have been implemented ordinarily as part of business as usual. In other words, neither A nor B by themselves would need a prior justification as to why they were deployed, and if the reasoning behind either of them was to be disclosed to the subjects, they would find them indistinguishable from any other business decision.
Of course, this argument would not apply if the A/B test involved any sort of artificial inconvenience (e.g. mock errors or delays) applied to either of the test conditions. I only mean A/B tests designed to compare features or behaviours which could both legitimately be considered beneficial, but the business is ignorant of which.
Yes, especially for critical projects?
> Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
I don’t jump to the conclusion that the random contributor is evil. I do however think about the potential impact of the submitted patch, security or not, and I do assume a random contributor can sneak in subtle bugs, usually not intentionally, but simply due to a lack of understanding.
>
> Yes, especially for critical projects?
People don't act that way I described intentionally, or because they are dumb.
Even if you go in with the greatest paranoia and the best of intentions, most of the time, most of the other people don't act maliciously and your paranoia eventually returns to a reasonable level (i.e. assuming that most people might not be malicious, but also not infallible).
It's a kind of fatigue. It's simply human. No matter how often you say "DUH of course they should".
In my entire life, I have only met a single guy who managed to keep that "everybody else is potentially evil" attitude up over time. IIRC he was eventually prescribed something with Lithium salts in it.
I’m not a maintainer but naively I would have thought that the answer to this is “Yes”.
I didn’t mean any disrespect. I didn’t write “I can’t believe they haven’t implemented a perfect technical process that fully prevents these attacks”.
I just asked if there are any ideas being discussed.
Two things can be true at the same time: 1. What the “researchers” did was unethical. 2. They uncovered security flaws.
Do the game theory. If you do assume that, you'll always be wrong. But if you don't assume it, you won't always be right.
People want to claim these are lone rogue researchers and good people at the university shouldn't be punished, but this is the only way you can reign in these types of rogues individuals: by getting the collective reputation of the whole university on the line to police their own people. Every action of individual researchers must be assumed to be putting the reputation of the university as a whole on the line. This is the cost of letting individuals operate within the sphere of the university.
Harsh, "over reaction" punishment is the only solution.
So yes.
Assuming this isn't being asked as a rhetorical question, I think that's exactly what turned the now infamous Facebook A/B test into a perceived unethical mass manipulation of human emotions. A lot of folks are now justifiably upset and skeptical of Facebook (and big tech) as a result.
So to answer your question: yes, if that test moves into territory that would feel like manipulation once the subject is aware of it. Maybe especially so because users are conceivably making a /choice/ to use said product and may switch to an alternative (or simply divest) if trust is lost.
Researchers at a company could arguably be deemed as engaging in unethical research and barred from contributing to the scientific community due to unethical behavior. Even doing experiments on your kids may be deemed crossing the line.
The question I have is when does it apply. If you research on your own kids but never publish, is it okay? Does the act of attempting to publish results retroactively make an experiment unethical? I'm not certain these things have been worked out because of how rare people try to publish anything that wasn't part of an official experiment.
The highest level is what had to be tested as well, or do you imagine only consulting Linus? Do you think that wouldn't've gotten him lynched?
There are experiments and experiments. Apart from the fact that they provided the fix right away, they didn’t do anyone harm.
And, by the way, it’s their job. Maintainers must approve patches after they ensured that the patch is fine. It’s okay to do mistakes, but don’t tell me “you’re wasting my time” after I showed you that maybe there’s something wrong with the process. If anything, you should thank me and review the process.
If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
Several of the patches are claimed to have landed in stable. Also, distributions and others (like the grsecurity people) pick up lkml patches that are not included in stable but might have security benefits. So even just publishing such a patch is harmful. Also, fixes were only provided to the maintainers privately as it seems, and unsuccessfully. Or not at all.
> If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
Exactly the same way as without that "research".
If you try to pry open my car door, I'll drag you to the next police station. "I'm just researching the security of car doors" won't help you.
I think people should be informed when market research is being done on them.
For situations where they are already invested in the situation, it should be optional.
For other situations, such as new customer acquisition, the person would have the option of simply leaving the site to avoid it.
But either way, they should be informed.
Yes please.
>> https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.... "We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected. The following shows the specific procedure of the experiment"
That's more of a story than what the researchers have done...
Btw .. I posted a few times on the thread, and want to acknowledge that researchers are humans, and humans do make mistakes. Thankfully in this case, the direct consequence was time wasted, and this is a teaching moment for all involved. In my humble opinion, the researchers should acknowledge in stronger terms they screwed up, do a post-mortem on how this happened, and everyone (including the researchers) should move on with their lives.
https://academia.stackexchange.com/questions/732/why-dont-re.... seems to list out reasons why not to do postmortems
I'm a little salty because I personally had two papers rejected by Oakland on the primary concern that their conclusions were too obvious already. I'd expect everybody to already believe that it wouldn't be too hard to sneak vulns into OSS patches.
And insists it was not human research. [1]
How can this type of people be professors?
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
And if this escalate to MSM Media it might also damage future employment status from UMN CS students.
Edit: Looks like they made a statement. https://cse.umn.edu/cs/statement-cse-linux-kernel-research-a...
- Signed by “Loren Terveen, Associate Department Head”, who was a co-author on numerous papers about experimenting on Wikipedia, as pointed out by: https://news.ycombinator.com/item?id=26895969
Edit: Parent comment originally referenced the paper that caused this mess.
As far as I'm concerned this university and all of its alumni are radioactive.
Your logic doesn't make ANY sense.
It is unfair to judge a whole university for the behavior of a professor or a department. Although I'm far from having all the details, it looks to me like the university is taking the right measures to solve the problem, which they acknowledge. I would understand your position if they tried to hide this or negated it, but as far as I understood that's not the case at all. Did I miss something?
And any piss I find, i will blame on amazon
Being a public university, I hope at some point they address this publicly as well as list the steps they are (hopefully) taking to ensure something like this doesn't happen again. I'm also not sure how they can continue to employ the prof in question and expect the open source community to ever trust them to act in good faith going forward.
The way I've seen Harvard, Stanford, and a few other university researchers dodge IRB review is by doing research in "private" time in collaboration with a private entity.
There is no effective oversight over IRBs, so they really range quite a bit. Some are really stringent and some allow anything.
how it does?
If they do so, the maintainers become more vigilant and the experiment fails. But, the key to the experiment is that maintainers are not vigilant as they should be. It’s not an attack to the maintainers though, but to the process.
A red team without approval is just a group of criminals. They must have been able to find active projects with a centralized leadership they could ask for permission.
And well, if the maintainers become more vigilant in the long run it's a win/win in my book.
And also, given that the stakes are higher e.g. in medicine, and the bar is lower in biology, one often gets a pass: "You don't want to poke anyone with needles, no LSD and no cages? Why are you asking us then?" Or something to that effect. The IRBs are just not used to such "harmless" things not being justified by the research objective.
The objectionable part is that the group allegedly continued after having been told to stop by the kernel developers.
In typical grey hat research you get pre-approval from target company leadership (engineers don't know) to avoid charges once discovered.
If you read the research paper linked in the lkml post, the authors at UMN state that they submitted their research plan to the University of Minnesota Institutional Research Board and received a human subjects exempt waiver.
Yeah, for one thing, to be a good analogy, rather than lockpicking without entering when he’s not home and leaving a note, you’d need to be an actual service worker for a trusted home service business and use that trust to enter when he is home, conduct sabotage, and not say anything until the sabotage is detected and traced back to you and cited in his cancelling the contract with the firm for which you work, and then cite the “research” rationale.
Of course, if you did that you would be both unemployed and facing criminal charges in short order.
Given that the professor appears to be a frequent flyer with this, the kernel folks banning him and the university prohibiting him from using Uni resources for anything kernel related seems reasonable and gets the point across.
You can’t make a silk purse out of a sow’s ear.
The paper you’re referring to was from last year. Two of the three patches that they emailed in under fake author names were rejected; they wrote a paper about the experience. All that happened as a result was that everybody told them that it was a terrible idea, and they tweaked the wording of the paper a bit.
Now _this_ year, a different PHD student with the same advisor posted a really dubious patch which would introduce one or more use–after–free bugs. This patch was also rejected by the maintainers. Greg noticed that it looks like another attempt to do the same kind of experiment again. Nobody but them know if that’s true or not, but the student reacted by calling it “slander”, which was not very advisable.
The methodology in the original paper had one redeeming feature; after any patch was accepted, they would immediately email back withdrawing the patch. That doesn’t appear to have happened in this case, but then this patch was rejected.
As a result of this, all future contributions from people affiliated with UMN are being rejected, and all past contributions (about 250) are being reviewed. Most of those are simply being backed out wholesale, unless someone speaks up for individual changes. A handful of those changes have already been vouched for.
That is pretty drastic, because there will certainly be acceptable patches that will need to be re–reviewed and possibly recommitted. On the other hand, if you discover a malicious actor, wouldn’t you want to investigate everything they’ve been involved with? On the gripping hand, there are such things as autoimmune diseases.
I guess we’ll have to see how it plays out.
There's no other option when someone on the same research team later sends them 4 diffs, 3 of which have security holes, than to assume they're still doing research in the same area.
This is what happens when you do a social experiment without at least informing someone in the organization beforehand. There's no way to verify whether it was well intentioned diffs or not. So you must assume it's not.
> IRB exempt was issued
https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
I agree this whole thing paints a really ugly picture, but it seems to validate the original concerns?
> You, and your group, have publicly admitted to sending known-buggy patches to see how the kernel community would react to them, and published a paper based on that work. > > Now you submit a new series of obviously-incorrect patches again, so what am I supposed to think of such a thing?
If they kept doing it even after being caught, is beyond understandable.
Edit: I am not defending the researchers who may have misled the IRB, or the IRB who likely have little understanding of what is actually happening
Meta-meta-experiment: submit the proposal above for IRB review and see what happens.
However, given that, my interpretation of the federal common rule is that this work would indeed fit the definition of human subjects research, as it comprises an intervention, and it is about generalizable human procedures, not the software artifact.
One institution I worked with conflated “exempt” and “not human subjects research” and required the same review of both.
Another institution separated them and would first establish if something was human subjects research. If it was, they would then review whether it was exempt from irb review based on certain categories. If they determined it was not human subjects research they would not review whether it met the exempt criteria, because in their mind they could not make such a determination for research that did not involve human subjects
The type of deception that is allowable in such cases is lying to participants about what it is that is being studied, such as telling people that they are taking a knowledge quiz when you are actually testing their reaction time.
Allowable deception does not include invading the space of people who did not consent to be studied under false pretenses.
It sounds pretty callous if that jet engine gets mounted on a plane that carries humans. In this hypothetical the IRB absolutely should have a hand in stopping research that has a methodology that includes sabotaging a jet engine that could be installed on a passenger airplane.
Waiving it off as an inanimate object doesn't feel like it captures the complete problem, given that there are many safety critical systems that can depend on the inanimate object.
> ... we don't need to do all the above and waste our time to fill some bureaucratic forms with unclear timelines and results. Our solution to ignore all @umn.edu contributions is much more reliable to us who are suffering from these researchers.
> ... we have the ability to easily go back and rip the changes out and we can slowly add them back if they are actually something we want to do.
> I will be working with some other kernel developers to determine if any of these reverts were actually valid changes, were actually valid, and if so, will resubmit them properly later. ... future submissions from anyone with a umn.edu address should be by default-rejected unless otherwise determined to actually be a valid fix
https://lore.kernel.org/lkml/YIBBt6ypFtT+i994@pendragon.idea...
> These are two different projects. The one published at IEEE S&P 2021 has > completely finished in November 2020. My student Aditya is working on a new > project that is to find bugs introduced by bad patches. Please do not link > these two projects together. I am sorry that his new patches are not > correct either. He did not intentionally make the mistake.
https://lore.kernel.org/lkml/YIBBt6ypFtT+i994@pendragon.idea...
The best analogy I could come with so far is; someone offered you compelling job offer, and when you where ready to sign up they would be yeah, that was research project, sorry - would you be ok with such behavior ?
This not ok, because you did not consent to waste your time on someone else's research project.
What is the point of dubbing yourself the arbiter of the moral high ground and spreading mis-information in the very next breath?
I am less puzzled by you spreading misinformation than I am by the fact you have this outrage at the very thing you are doing and don't hesitate to attack the character of people you disagree with.
> A number of these patches they submitted to the kernel were indeed successfully merged to the Linux kernel tree.
It turns out the researchers DID allow the bad faith commits to be merged and that is a big problem that is still being unwound.
https://fosspost.org/researchers-secretly-tried-to-add-vulne...
But you also forgot the part where Greg throws a hissy fit and decides to revert every commit from umn emails, including 3+year old commits that legitimately fix security vulns.[0] Great job keeping mainline bug free with your paranoid witchhunt!
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
If you know of any others that shouldn’t be reverted, you should email the list and point them out.
* Is this human research? This is not considered human research. This project studies some issues with the patching process instead of individual behaviors, and we did not collect any personal information. We send the emails to the Linux community and seek community feedback. The study does not blame any maintainers but reveals issues in the process. The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained).
I’m not saying it is okay, I’m simply saying how this could happen.
It requires understanding the connection between inanimate object and personal harm, which in this case is 1)non obvious and 2)not even something I necessarily accept within a common rule definition of harm.
Annoyance or inconvenience is not a meaningful human harm within the irb framework
But, fundamentally, the irb did not see this as human research. You and I and the commenters see how that is wrong. That is where their evaluation ended...they did not see human involvement right or wrong.
And irb is part of the discussion of research ethics, it is not the beginning nor the end of doing ethical research.
https://en.wikipedia.org/wiki/Peter_Boghossian#Research_misc...
* The researcher is a professor in the humanities, which typically does not deal with human subjects research and the (often) vague and confusing boundaries. Often, people from outside the social sciences and medical/biology fields struggle a little bit with IRBs because...things don't seem rational until you understand the history and details. Just like someone from CS.
* The researcher in your example DID NOT seek review by IRB (per my memory of the situation). That was the problem. The kernel bug authors seem to have engaged with their IRB. the difference is not doing it vs. a misunderstanding.
* The comments about seeking consent before submitting the fake papers ignore that it is perfectly possible to have done this WITHOUT a priori informed consent. It is perfectly possible for IRBs to review and approve studies involving deception. In those cases, informed consent is not required to collect data.
* Finally, people on IRBs tend to be academics and are highly likely to have some understanding of how a journal works. That would mean they understand the human role in journal publishing. The exact same IRB may well not have anyone with CS experience and may have looked at the kernel study and seen the human role differently than journal study.
* Lastly, the fact that the IRB in your example looked at 'animal rights' is telling. They were trying to figure out what Peter did. He published papers with data about experiments on animals...that would require IRB review. The fact that that charge was dismissed when they figured out no such experiments occurred is telling about who is acting in good faith.
Do you think that the IRB was correct to make the determination they did? It does sound like a bit of a grey area
> The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained). Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning.
So the statement is a bit unclear to me, and I’m hesitant to come to a conclusion because I have not seen what they submitted.
As I read this they are saying:
* we explained the study to irb and asked whether it met their definition of human subjects research - based on our description they said it is not human subjects research
* therefore we did not apply to irb to have the study assessed for the appropriate type of review.
Exempt is a type of irb review, basically it is a low level desk review of a study. It does not mean no one looks at it, it just means the whole irb doesn’t have to discuss it.
I can see both sides of this. Irbs focus on protection of the rights of research participants. The assumption in cognitive models is of direct participants. This study ended up having indirect participants. I would argue that is the researchers job to clarify and ensure was reviewed. However, there is almost certainty this study would have been approved as exempt.
I think the irb likely did the right thing based on the information provided to them. The harm that HN is identifying does not fall within the normal irb definitions of harm anyways...which is direct harm to people. The causal chain HN is spun up about is very real...just not how irb views research typically
I don't think that was the conclusion.
There is no such thing as a process that can compensate for trust mechanisms. Or if you want to view it that way, ignoring the university's protests and blanket-banning all contributions made by anybody there with no further investigation is part of the process.
What this means is that anyone who could hijack a university email account, or could be a student at a state university for a semester or so, or work at a FAANG corporation could pretty much insert backdoors without a lot of scrutiny in a way that no one detects, because there aren't robust safeguards in place to actually verify that commits don't do anything sneaky beyond trusting that everyone is acting in good faith because of how they act in a code review process. I have trouble understanding the thought process that ends up basically ignoring the maintainers' duty to make sure that the code being committed doesn't endanger security or lives because they assumed that everything was 'cool'. The security posture in this critical infrastructure is deficient and no one wants to actually address it.
He shouldn't be surprised if it has some unexpected consequences to his own personal security, like some unknown third parties porting away his phone number(s) as a social engineering test, pen testing his office, or similar.
I would guess their IRB had a quick sanity check process to ensure there was no human subject research in the experiment. This is actually a good thing if scientists use their ethics and apply good judgement. Now, whoever makes that determination does so based on initial documentation supplied by the researchers. If so, the researchers should show what they submitted to get the exemption.
Again, the implication is their University will likely make it harder to get exemptions after this fiasco. This mistake hurts everyone (be it indirectly). Although, and this is being quite facetious and macabre, the researchers have inadvertently exposed a bug in their own institutions IRB process!
I hope they release what they submitted to the IRB to receive that exemption and there are some form of consequences if the mistake is on their part.
1. You have to submit for review any work involving human subjects before you start interacting with them. The authors clearly state that they sought retroactive approval after being questioned about their work. That would be a big red flag for my IRB and they wouldn't approve work retroactively.
2. There are multiple levels of IRB approval. The lowest is non regulated, which means that the research falls outside of human subject research. Individual researchers can self-certify work as non regulated or get a non-regulated letter from their IRB.
From there, it goes from exempt to various degrees of regulated. Exempt research means that it is research involving human subjects that is exempt from continued IRB review past the initial approval. That means that IRB has found that their research involves human subjects but falls within one (or more) of the exceptions for continued review.
In order to be exempt, a research project must meet one of the exemptions categories (see here https://hrpp.msu.edu/help/required/exempt-categories.html for a list). The requirements changed in 2018, so what they had to show depends on when they first received their exempt status.
The bottom line is that the research needs to (a) have less than minimal risks for participants and (b) needs to be benign in nature. In my opinion, this research doesn't meet these requirements as there are significant risks to participants to both their professional reputation and future employability for having publicly merged a malicious patch. They also pushed intentionally malicious patches, so I am not sure if the research is benign to begin with.
3. Even if a research project is found exempt from IRB review, participants still need to consent to participate in it and need to be informed of the risks and benefits of the research project. It seems that they didn't consent their participants before their participation in the research project. Consent letters usually use a common template that clearly states the goals for the research project, lists the possible risks and benefits of participating in it, states the name and contact information of the PI, and data retention policies. IRB could approve projects without proactive participant consent but those are automatically "bumped up" to full IRB approval and approvals are given only in very specific circumstances. Plus, once a participant removes their consent to participate in a research project, the research team needs to stop all interactions with them and destroy all data collected from them. It seems that the kernel maintainers did not receive the informed consent materials before starting their involvement with the research project and have expressed their desire not to participate in the research after finding out they were participating in it, so the interaction with them should stop and any data collected from them should be destroyed.
4. My impression is that they got IRB approval on a technicality. That is, their research is on the open source community and its processes rather than the individual people that participate in them. My impression of their paper is that they are very careful in addressing the "Linux community" and they really never talk about their interaction with people in the paper (e.g., there is no data collection section or a description of their interactions on the mailing list). Instead, it's my impression that they present the patches that they submitted as happening "naturally" in the community and that they are describing publicly available interactions. That seems to be a little misleading of what actually happened and their role in producing and submitting the patches.
What if they did a survey of passers–by on a public street, that might be in view of CCTV operated by someone else?
Notify someone up the chain that you want to submit malicious patches, and ask them if they want to collaborate.
If your patches make it through, treat it as though they essentially just got red teamed, everyone who reviewed it and let it slip gets to have a nervous laugh and the commit gets rejected, everyone having learned something.
From the looks of it they didn't even when it was heading out to stable releases?
That's just using the project with no interest in not causing issues.
The patches were merged and the email thread discusses that the patches made it to the stable tree. Some (many?) distributions of Linux have and run from stable.
> 2. There are mechanisms in place for preventing actual merging of the faulty patches.
Those mechanisms failed.
> 3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
Arguably. But I think this is a weak argument.
The approved methodology - described in the linked paper - was that when a patch with the introduced vulnerabilities is accepted by its reviewer, the patch submitter indicates that the patch introduces a vulnerability exists, and sends a no-vulnerability version. That's what the paper describes.
If the researchers did something other than what the methodology called for (and what the IRB approved), then perhaps the analogy may be valid.
It's irrelevant whether any bugs were ultimately introduced into the kernel. The fact is the researchers deliberately abused the trust of other human beings in order to experiment on them. A ban on further contributions is a very light punishment for such behavior.
This analogy is pretty valid, the in-flight-experiment analogy is invalid.
The important thing is not to hunt for motives but to identify and quarantine the saboteurs to prevent further sabotage. Complaining to the University's research ethics board might help, because, regardless of intent, sabotage is still sabotage, and that is unethical.
"Dear GK-H: I would like to have my students test the security of the kernel development process. Here is my first stab at a protocol, can we work on this?"
and
"We're going to see if we can introduce bugs into the Linux kernel, and probably tell them afterwards"
is the difference between white-hat and black-hat.
Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
To adress your first critic: benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm. Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
That's an assertion. A hypothesis is verified through observing the real world. You can do that in many ways, giving you different confidence levels in validity of the hypothesis. Research such as the one we're discussing here is one of the ways to produce evidence for or against this hypothesis.
> Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
It is if there's a review process. Auditability itself is really most interesting before a patch is accepted. Sure, it's nice if vulnerabilities are found eventually, but the longer that takes, the more likely it is they were already exploited. In case of an intentionally bad patch in particular, the window for reverting it before it does most of its damage is very small.
In other words, the experiment wasn't testing the entire auditability hypothesis. Just the important part.
> benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm
Sure. But the project scope matters. Linux kernel isn't some random OSS library on Github. It's core infrastructure of the planet. Assumption of benevolence works as long as the interested community is small and has little interest in being evil. With infrastructure-level OSS projects, the interested community is very large and contains a lot of malicious actors.
> Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
I agree, and in my books, if a legitimate researcher gets banned for such "undercover" research, it's just the flip side of doing such experiment.
Before a patch is accepted, "auditability" is the same in OSS vs in proprietary, because both pools of engineers in the review groups have similar qualifications and approximatively the same number of people are involved.
So, the real advantage of OSS is on the auditability after the patch is integrated.
While any USB stick might have malware on it if it's ever been out of your sight, that one you found in the parking lot is a much bigger problem.
Sabotage is a very real risk but we're discussing ethics of demonstrating the risk instead of potential remediation, that's dangerous and foolish.
It really doesn't though. You can claim ownership of that email address in the published manuscript. For that matter, you could even publish the academic article under a pen name if you wanted to. But after seeing how the maintainers responded here, you'd better make sure that any "real" contributions you make aren't associated with the activity in any way.
You're criticizing the process, but the truth is that without a real name email and an actual human being's "social credit" to be burned, there's no proof these researchers would have achieved the same findings. The more interesting question to me is if they had used anonymous emails, would they have achieved the same results? If so, there might be some substance to your contrarian views that the process itself is flawed. But as it stands, I'm not sure that's the case.
Why? Well, look at what happened. The maintainers found out and blanket banned bad actors. Going to be a little hard to reproduce that research now, isn't it? Arbitraging societal trust for research doesn't just bring ethical challenges but /practical/ ones involving US law and standards for academic research.
And as unfortunate as it sound it look like all victim of such generalization, the alumni would have to fight the prejudice associated to their choice of university.
https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
https://grants.nih.gov/policy/humansubjects/hs-decision.htm
for q1, the study collects data like observations of behavior, so we must answer yes.
for q2, none of the exemptions apply - it's not an educational setting, they're not sending a survey, it's not an observation of the public - they're interacting, and it's clear that these interactions are not benign - they have clear impact on the community. None of these exemptions apply.
Based on this flow, it's clear the study involves "human research".
If they have a problem in mastering English, they can take lessons, and make native speaker review their communication in the meantime.
The benefit of the doubt can not stick for ever on people caught red-handed. It can be restored of course, but they are now in a position where they drastically shifted the perception by their own actions, and thus can't really complain of the results of their own doings. Yes, they can not make mistakes anymore, and everything they did in the past will be reviewed harshly, not for further condemning them without reasons, but just to be sure they did not actually break things while practicing their malicious activities.
If I can come up with the scientific paper gibberish for that in real-time, and I don’t even write science papers, then these people who understand how to navigate an ethical review board process surely know how to massage an unpleasant truth into dry and dusty wording.
I think that they just screwed up and missed the word “precious” in editing, and thus got caught being dismissive and snide towards their experiment’s participants. Without that word, it’s a plausible enough paragraph. With it, it’s no longer plausibly innocent.
> In the past several years, we devote most of our time to improving the Linux kernel, and we have found and fixed more than one thousand kernel bugs; the extensive bug finding and fixing experience also allowed us to observe issues with the patching process and motivated us to improve it. Thus, we consider ourselves security researchers as well as OSS contributors. We respect OSS volunteers and honor their efforts.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
It feels to me you have jumped to an untenable conclusion without even considering their point of view.
The fact that a mailing list is publicly available is what made me worry about the applicability of any sort of exemption. In order for human subject research to be exempt from IRB review, the research needs to be deemed less than minimal risk to participants.
The fact that their experiment happens in public and that anyone can find their patches and individual maintainers' responses (and approval) of them makes me wonder if the participants are at risk of losing professional reputation (in that they approved a patch that was clearly harmful) or even employment (in that their employer might find out about their participation in this study and move them to less senior positions as they clearly cannot properly vet a patch). This might be extreme, but it is still a likely outcome given the overall sentiment of the paper.
All research that poses any harm to participants has to be IRB approved and the researchers have to show that the benefits to participants (and the community at large) surpass the individual risks. I am still not sure what benefits this work has to the OSS community and I am very surprised that this work did not require IRB supervision at all.
As far as work on a public street is concerned, IRB doesn't regulate common activities that happen in public and for which people do not have a reasonable expectation of privacy. But, as soon as you start interacting with them (e.g., intervene in their environment), IRB review is required.
You can read and analyze a publicly available mailing list (and this would even qualify as non human subject research if the data is properly anonymized) without IRB review or at most a deliberation of exempt status but you cannot email the mailing list yourself as a researcher as the act of emailing is an intervention that changes other people's environment, therefore qualifying as human subject research.
They're banning a group known to be bad actors. And proactively tearing out the history of commits related to those known actors, before reviewing each commit.
That seems like the kernel team are taking a proactive stance on the security side of this. The LKML thread also talks about more stringent requirements that they're going to bring in, which was already going to be brought up at the next kernel conference.
None of these things seem like ignoring any of the security issues.
Besides, are you arguing that ends justify the means if the intent behind the research is valid?
It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused. There's no point doing this, you can just search Wikipedia's edits for corrections, and start your analysis from there.
Isn't this part still experimenting on people without their consent? Why does one group of maintainers get to decide that you can experiment on another group?
Does creating a vaccine justify the death of some lab animals? Probably.
Does creating supermen justify mutilating people physically and psychologically without their consent? Hell no.
You can’t just ignore the context.
That has the risk that the contacted maintainer is later accused of collaborating with saboteurs or that they consult others. Either very awful or possibly invalidates results.
> 2) Create a group of maintainers who know the experiment is going to happen, but leave a certain portion of the org out of it
Assuming the leadership agrees and won't break confidentiality, which they might if the results could make them look bad. Results would be untrustworthy or potentially increase complacency.
> 4) Interfere before any further damage is done
That was done, was it not?
> Besides, are you arguing that ends justify the means if the intent behind the research is valid?
Linux users are lucky they got off this easy.
In this case, in my opinion, a small set of maintainers and linus as "management" would have to be in the know to e.g. stop a merge of such a patch once it was accepted by someone in the dark.
Kernel maintainers are volunteering their time and effort to make Linux better, not to be entertaining test subjects for the researchers.
Even if there is no ethical violation, they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
Given the importance of the Linux kernel, there has to be a way to make contributions safer. Some people even compare it to the "water supply" and others bring in "national security".
> they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
"Oh no, think of the effort we have to spend at defending a critical piece of software!"
How are kernel maintainers competent in detecting a real person vs. fake real person? Why is there any inherit trust?
It's clear the system is fallible, but at least now people are humbled enough to not instantly dismiss the risk.
> The maintainers found out and blanket banned bad actors.
With collateral damage.
What would you like them to do instead or in addition to this?
Update the processes and tools to try and catch such malicious infiltrators. Lynching researchers isn't fixing the actual issue right now.
How?
Yeah, there’s a reason the US response to 9/11 wasn’t to name Osama bin Laden “Airline security researcher of the Millenium”, and it isn’t that “2001 was too early to make that judgement”.
Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet.
The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.
I trust Greg to have not edited or misconstrued their response.
https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
Perhaps the researchers see no harm in letting that be released.
It might be less intentionally harmful if we presume they didn't know other patches introduced vulnerabilities, but this is also why this research methodology is extremely reckless and frustrating to read about, when this could have been done with guard rails where they were needed without impacting the integrity of the results.
The final commit in the reverted list (d656fe49e33df48ee6bc19e871f5862f49895c9e) is originally from 2018-04-30.
EDIT: Not all of the 190 reverted commits are obviously malicious:
https://lore.kernel.org/lkml/20210421092919.2576ce8d@gandalf...
https://lore.kernel.org/lkml/20210421135533.GV8706@quack2.su...
https://lore.kernel.org/lkml/CAMpxmJXn9E7PfRKok7ZyTx0Y+P_q3b...
https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
What a mess these guys have caused.
The submitter has to remember to send the "warning, don't apply patch" mail in the short time window between confirmation and merging. What happens if one of the students doing this work gets sick and misses some days of work, withdraws from the program, just completely forgets to send the mail?
What if the reviewer doesn't see the mail in time or it goes to spam?
In short, yes. Every attempted defense of them has operated by taking their statements at face value. Every position against them has operated by showing the actual facts.
This may be shocking, but there are some people in this world who rely on other people naively believing their version of events, no matter how much it contradicts the rest of reality.
I think they are saying that it's possible that some code was branched and used elsewhere, or simply compiled into a running system by a user or developer.
Anyway, my point wasn't that this is free of ethical concerns, but it seems like they put _some_ thought into how to reduce the potential harm. I'm undecided if that's enough.
I don't think it's anywhere close to enough and I think their behavior is rightly considered reckless and unethical.
They should have contacted the leadership of the project to announce to maintainers that anonymous researchers may experiment on the contribution process, allowed maintainers to opt out, and worked with a separate maintainer with knowledge of the project to ensure harmful commits were tracked and reversions were applied before reaching stable branches.
Instead their lack of ethical considerations throughout this process has been disappointing and harmful to the scientific and open source communities, and go beyond the nature of the research itself by previously receiving an IRB exemption by classifying this as non-human research, and potentially misleading UMN on the subject matter and impact.
https://www.startribune.com/markingson-case-university-of-mi...
When someone graduates from the university, that is the same as the university saying "This person is up to our standards in terms of knowledge, ethics and experience."
If those standards for ethics are very low, then it naturally taints that reputation they sold.
Sometimes you might need to make the committee understand before a full review when you are asking where a line is for some tricky part, but you ask about those parts long before you have enough of the study designed to actually put it before the review.
Ethics are a personal responsibility. You should be personally embarrassed if you ever have something fail review, and probably should have your tenure removed as well since if your ethics are so bad as to put before the board something that fails you will also do something even worse without any review.
I submit unclear thing
Thing is approved
Thing must have been clear right?
2. Something being grounds for expulsion and what a reasonable response would be are vastly different things.
3. The rules saying "up to and including" (aka grounds for) and the full range of punishment are not the same - the max of a range is not the entirety of the range.
4. So what?
That's very true. It's worth noting that various legal and security tools deployed by the society help us understand what are the real limits to "mostly".
So for example, the cryptocurrency crowd is very misguided in their pursuit of replacing trust with math - trust is the trick, the big performance hack, that allowed us to form functioning societies without burning ridiculous amounts of energy to achieve consensus. On the other hand, projects like Linux kernel, which play a core role in modern economy, cannot rely on assumption of benevolence alone - incentives for malicious parties to try and mess with them are too great to ignore.
We live in a society, to operate open communities there are trade-offs.
If you want to live in a miserable security state where no action is allowed, refunds are never accepted, and every actor is assumed hostile until proven otherwise, then you can - but it comes at a serious cost.
This doesn't mean people shouldn't consider the security implications of new PRs, but it's better to not act like assholes with the goal being a high-trust society, this leads to a better non-zero-sum outcome for everyone. Banning these people was the right call they don't deserve any thanks.
In some ways their bullshit was worse than a real bad actor actually pursuing some other goal, at least the bad actor has some reason outside of some dumb 'research' article.
The academics abused this good-will towards them.
What did they show here that you can sneak bugs into an open source project? Is that a surprise? Bugs get in even when people are not intentionally trying to get them in.
I strive for a high trust society too. Totally agree. And acknowledging that people can exploit trust and use it to push poor code through review does not dismantle a high trust operation or perspective. Trust systems fail when people abuse trust so the reality is that there must be safeguards built in both technically and socially in order to achieve a suitable level of resilience to keep things sustainable.
Just look at TLS, data validation, cryptographic identity, etc. None of this would need to exist in a high trust society. We could just tell people who we are, trust other not to steal our network traffic, never worry about intentionally invalid input. Nobody would overdraft their accounts at the ATM, etc. I find it hard to argue for absolute removal of the verify step from a trust but verify mentality. This incident demonstrated a failure in the verify step for kernel code review. Cool.
There is further indication that the patches to revert are not mostly/not at all vulnerability-introducing patches in a message by "Steve" which says:
> The one patch from Greg's reverts that affects my code was actually a legitimate fix
So, again, while it is still theoretically possible that vulnerabilities were introduced into stable, that is not known to be the case.
If that's the claim, then the research work discussed here is indeed not relevant to it.
But also, if that's the claim, then it's easy to point out that the "advantage" here is hypothetical, and not too important in practice. Most people and companies using OSS rely on release versions to be stable and tested, and don't bother doing their own audit. On the other hand, intentional vulnerability submission is an unique threat vector that OSS has, and which proprietary software doesn't.
It is therefore the window between patch submission and its inclusion in a stable release (which may involve accepting the patch to a development/pre-release tree), that's of critical importance for OSS - if vulnerabilities that are already known to some parties (whether the malicious authors or evil onlookers) are not caught in that window, the threat vector here becomes real, and from a risk analysis perspective, negates some of the other benefits of using OSS components.
Nowhere here I'm implying OSS is worse/better than proprietary. As a community/industry, we want to have an accurate, multi-dimensional understanding of the risks and benefits of various development models (especially when applied to core infrastructure project that the whole modern economy runs on). That kind of research definitely helps here.
On this specific point, it only holds if you restrict the assertion to 'intentional submission of vulnerabilities by outsiders'. I don't work in fintech, but I've read allegations that insider-created vulnerabilities and backdoors are a very real risk.
Very fair point. Inside threat also exists in corporations, but it's probably harder.
The Tuskegee Study wouldn't have happened if its participants were voluntarily, and it's effects still haunt the scientific community today. The attitude of "science by any means, including by harming other people" is reprehensible and has lasting consequences for the entire scientific community.
However, unlike the Tuskegee Study, it's totally possible to have done this ethically by contacting the leadership of the Linux project and having them announce to maintainers that anonymous researchers may experiment with the contribution process, and allowing them to opt out if they do not consent, and to ensure that harmful commits never reach stable from these researchers.
The researchers chose to instead lie to the Linux project and introduce vulnerabilities to stable trees, and this is why their research is particularly deplorable - their ethical transgressions and possibly lies made to their IRB were not done out of any necessity for empirical integrity, but rather seemingly out of convenience or recklessness.
And now the next group of researchers will have a harder time as they may be banned and every maintainer now more closely monitors academics investigating open source security :)
This introduced security vulnerabilities to stable branches of the project, the impact of which could have severely affected Linux, its contributors, and its users (such as those who trust their PII data to be managed by Linux servers).
The potential blast radius for their behavior being poorly tracked and not reverted is millions if not billions of devices and people. What if a researcher didn't revert one of these commits before it reached a stable branch and then a release was built? Linux users were lucky enough that Greg was able to revert the changes AFTER they reached stable trees.
There was a clear need of informed consent of *at least* leadership of the project, and to say otherwise is very much in defense of or downplaying the recklessness of their behavior.
I acknowledged that lives are not at play, but that doesn't mean that the only consequence or concern here was wasting the maintainers time, especially when they sought an IRB exemption for "non-human research" when most scientists would consider this very human research.
It doesn't block patch submissions from students of professors using their private email, since that assumes they are contributing as individuals, and not as employees or students.
It's as close as practically possible to blocking an institution and not the individuals.
> As far as I'm concerned this university and all of its alumni are radioactive.
That is not a practical issue, but a too broad generalization (although, I repeat, I may have missed something).
It's the same as with employees. If I get a patch request from xyz@ibm.com I'll assume that it comes from IBM, and that person is submitting a patch on behalf of IBM, while for a patch coming from xyz@gmail.com I would not assume any IBM affiliation or bias, but assume person contributing as an individual.
Source:
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C22&q=Lor...
This is quite literally the first point of the Nuremberg code research ethics are based on:
https://en.wikipedia.org/wiki/Nuremberg_Code#The_ten_points_...
This isn't an individual failing. This is an institutional failing. This is the sort of thing which someone ought to raise with OMB.
He literally points to how Wikipedia needed to respond when he broke the rules:
https://en.wikipedia.org/wiki/Wikipedia:What_Wikipedia_is_no...
Doesn't mean there aren't ethical issues related to editors being human subjects, but you may want to be more specific.
What did you see that offended you?
I'm not sure how it affects things, but I think it's important to clarify that they did not obtain the IRB-exempt letter in advance of doing the research, but after the ethically questionable actions had already been taken:
The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained). Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. ... We would like to thank the people who suggested us to talk to IRB after seeing the paper abstract.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
That's not really what they did.
They sent the patches, the patches where either merged or rejected.
And they never let anybody knew that they had introduced security vulnerabilities on the kernel on purpose until they got caught and people started reverting all the patches from their university and banned the whole university.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> (4). Once any maintainer of the community responds to the email, indicating “looks good”, we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
Seems like there are some patches already on stable trees [1], so they're either lying, or they didn't care if those "don't merge" messages made anybody react to them.
1 - https://lore.kernel.org/linux-nfs/CADVatmNgU7t-Co84tSS6VW=3N...
>> The work taints the relationship between academia and industry
> We are very sorry to hear this concern. This is really not what we expected, and we strongly believe it is caused by misunderstandings
Yeah, misunderstandings by the university that anyone, ever, in any line of endeavor would be happy to be purposely fucked with as long as the perpetrator eventually claims it's for a good cause. In this case the cause isn't even good, they're proving the jaw-droppingly obvious.
Yes, that's the whole point! The real malicious actors aren't going to notify anyone that they're injecting vulnerabilities either. They may be plants at reputable companies, and they'll make it look like an "honest mistake".
Had this not been caught, it would've exposed a major flaw in the process.
> ...until they got caught and people started reverting all the patches from their university and banned the whole university.
Either these patches are valid fixes, in which case they should remain, or they are intentional vulnerabilities, in which case they should've already been reviewed and rejected.
Reverting and reviewing them "at a later date" just makes me question the process. If they haven't been reviewed properly yet, it's better to do it now instead of messing around with reverts.
While true, it's simply not acceptable to abuse trust in this way. It causes real emotional harm to real humans, and while it also may produce some benefits, those do not outweigh the harms. Just because malicious actors don't care about the harms shouldn't mean that ethical people shouldn't either.
Well, in real life, you can't go punch someone in the face to teach them a "point". If you do so, you'll get punished.
> Reverting and reviewing them "at a later date" just makes me question the process.
I don't think anybody realistically thought that the kernel review process is rock solid against malicious anyway. What exactly does the paper expose?
This just turns the researchers into black hats. They are just making it look like "a research paper."
How is this not human research? They experimented on the reactions of people in a non-controlled environment.
It is likely the professor involved here will be fired if they are pre-tenure, or sanctioned if post-tensure.
This research had the potential to cause harm to people despite not being human research and was therefore ethically questionable at best. Because they presented the research as not posing potential harm to real people that means they lied to the IRB, which is grounds for dismissal and potential discreditation of all participants (their post-graduate degrees could be revoked by their original school or simply treated as invalid by the educational community at large). Discreditation is unlikely, but loss of tenure for something like this is not out of the question, which would effectively end the professor's career anyway.
I don't buy it, and you fail to back that claim up at all.
It seems very possible to me that an IRB wouldn't have accepted their proposed methodology if they hadn't received an exemption.
Is there anyone on hand who could explain how what looks very much like a social engineering attack is not "human research"?
It's a specific threat model they were exploring: a malicious actor introducing vulnerability on purpose.
> Couldn't they just look at the history of security vulnerabilities in the kernel, and analyze how long it took for them to be detected?
Perhaps they could. I guess it'd involve much more work, and could've yielded zero results - after all, I don't think there are any documented examples when a vulnerability was proven to have been introduced on purpose.
> what's the point of all this subterfuge in the first place?
Control over the experimental setup, which is important for validity of research. Notice how most research involves gathering up fresh subjects and controls - scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for. They want fresh subjects to better account for possible confounders, and hopefully make the experiment reproducible.
(Similarly, when chasing software bugs, you could analyze old crash dumps all day to try and identify a bug - and you may start with that - but you always want to eventually reproduce the bug yourself. Ultimately, "I can and did that" is always better than "looking at past data, I guess it could happen".)
> It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused.
Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good. Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
Also note that both Wikipedia and Linux kernel are essentially infrastructure now. Running research like this against them makes sense, where running the same research against a random small site / OSS project wouldn't.
But does that matter? We can imagine that the error-prone developer who submitted the buggy patch just had a different mindset. Nothing about the patch changes. In fact, a malicious actor is explicitly trying to act like an error-prone developer and would (if skilled) be indistinguishable from one. So we'd expect the maintainer response to be the same.
In line with UncleMeat's comment, I'm not convinced it's of any consequence that the security flaw was introduced deliberately, rather than by accident.
> scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for
That doesn't sound like a fair description of what's happening here.
There are two things at play. Firstly, an analysis of the survival function [0] associated with security vulnerabilities in the kernel. Secondly, the ability of malicious developers to deliberately introduce new vulnerabilities. (The technical specifics detailed in the paper are not relevant to our discussion.)
I'm not convinced that this unethical study demonstrates anything of interest on either point. We already know that security vulnerabilities make their way into the kernel. We already know that malicious actors can write code with intentional vulnerabilities, and that it's possible to conceal these vulnerabilities quite effectively.
> Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good.
That's like saying It's ok to deface library books, provided it's a large library, and provided other people are also defacing them.
Also, it would not yield a social good. As I already said, it's possible to study Wikipedia's ability to repair vandalism, without committing vandalism. This isn't hypothetical, it's something various researchers have done. [0][1]
> Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
It isn't. Universities have ethics boards. They are held to a higher ethical standard, not a lower one.
> Running research like this against them makes sense
No one is contesting that Wikipedia is worthy of study.
[0] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...
[1] https://en.wikipedia.org/wiki/Wikipedia:Counter-Vandalism_Un...
I think "lamenting" is very much the wrong attitude here. Given all the things that make use of Linux today that seems like the only sane approach to me.
The allegation being made on the mailing list is that some incorrect patches of theirs made it into git and even the stable trees. As there is not presently an enumeration of them, or which ones are alleged to be incorrect, I cannot state whether this is true.
But that's the claim.
edit: And looking at [1], they have a bunch of relatively tiny patches to a lot of subsystems, so depending on how narrowly gregkh means "rip it all out", this may be a big diff.
edit 2: On rereading [2], I may have been incorrectly conflating the assertion about "patches containing deliberate bugs" with "patches that have been committed". Though if they're ripping everything out anyway, it appears they aren't drawing a distinction either...
[1] - https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
[2] - https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
[1] - https://lore.kernel.org/linux-nfs/YIAta3cRl8mk%2FRkH@unreal/
You can have your verify-lite process, but you must write down that that was your decision, and if appropriate, revisit and reaffirm it over time. You must implement controls, measures and processes in such a way as to minimize the deleterious consequences to your endeavor. It's the entire reason Quality Assurance is a pain in the ass. When you're doing a stellar job, everyone wonders why you're there at all. Nobody counts the problems that didn't happen or that you've managed to corral through culture changes in your favor, but they will jump on whatever you do that drags the group down. Security is the same. You are an anchor by nature, the easiest way to make you go away is to ignore you.
You must help, first and foremost. No points for groups that just add more filth to wallow through.
I'm not saying this is innocent, but it's not at all clear to me that vulnerabilities were deliberately introduced with the goal of allowing them to reach a release.
Anyway, like I said, too unclear for me to have an opinion.
While I'm sure a room of people might find it useful to psychoanalyze their 'unclear' but probably malicious intent, their actions are clearly harmful to researchers, Linux contributors, direct Linux users, and indirect Linux users (such as the billions of people who trust Linux systems to store or process their PII data).
Now, if this protocol were not followed, that's a different story, but we do not know this to be the case.
Of course, there are other ethical and legal requirements that you're bound to, not just this one. I'm not sure which requirements IRBs in the US look into though, it's a pretty murky situation.
Main point is that IRBs were created in response to some highly unethical and harmful "studies" being carried out by institutions thought of as top-tier. Now they are considered to be a mandatory part of carrying out ethical research. But if you think about it, isn't outsourcing all sense of ethics to an organization external to the actual researchers kind of the opposite of what we want to do?
All institutions tend to be corruptible. Many tend to respond to their actual incentives rather than high-minded statements about what they're supposed to be about. Seems to me that promoting the attitude of "well an IRB approved it, so it must be all right, let's go!" is the exact opposite of what we really want.
All things considered, it's probably better to have something there than nothing. But you still have to be responsible for your own decisions. I bamboozled our lazy IRB into approving our study, so I'm not responsible for it being obviously a bad idea, just isn't good enough.
If you think about it, it's actually kind of meta to the code review process they were "studying". Just like IRBs, Code review is good, but no code review process will ever be good enough to stop every malicious actor every time. It will always be necessary to track the reputation of contributors and be able to mass-revert contributions from contributors later determined to be actively malicious.
Eventually, the IRB, unhappy at his behavior, said he couldn't do the experiment. He left for another institution (UC San Diego) immediately, having made a deal with the new dean to go through expedited review. It was a big loss for Boulder and TBH, the IRB's reasoning was not sound.
That's not what the comment I was responding to said. It was very clear: "As far as I'm concerned this university and all of its alumni are radioactive". It does not say every kernel patch coming from this domain is radioactive, it clearly says "all of its alumni are radioactive".
You said before that alumni from the university could submit patches with their private emails, but according to what djbebs said, he would not. Do we agree that this would be wrong?
It's not about guilt, it's about trust. They were trained for years in an institution that violates trust as a matter of course. That makes them suspect and the judgement completely fair.
"As a matter of course" is a big leap here.
you think students believe in everything that profs do/say?
Anonymity is de facto, not de jure. It's also a privilege for many collaboration networks and not a right. If abused, it will simply be removed.
Given what the Linux kernel runs these days, that would probably be advisable. (I'm a strong proponent of anonymity, but I also have a preference that my devices not be actively sabotaged.)
> we're entering the territory of fraud and cybercrime
So what? The fact that it's illegal doesn't nullify the threat. For that matter, it's not even a crime if a state agency is the perpetrator. These researchers drew attention to a huge (IMO) security issue. They should be thanked and the attack vector carefully examined.
That is exact opposite of how rot in literal bunch of apples behave. Spoil spreads throughout the whole lot very, very quickly.
The chief problem here is not that it bruises the egos of the Linux developers for being psyched, but that it was a dick move whereby people now have to spend time sorting this shit out.
Prof Liu miscalculated. The Linux developers are not some randos off the street where you can pay them a few bucks for a day in the lab, and then they go away and get on with whatever they were doing beforehand. It's a whole community. And he just pissed them off.
It is right that Linux developers impose a social sanction on the perpetrators.
It has quite possibly ruined the student's chances of ever getting a PhD, and earned Liu a rocket up the arse.
I disagree. I think it's easier to excuse bad judgment, in part because we all sometimes make mistakes in complicated situations.
But this is an example of experimenting on humans without their consent. Greg KH specifically noted that the developers did not appreciate being experimented on. That is a huge chasm of a line to cross. You are generally required to get consent before experimenting on humans, and that did not happen. That's not just bad judgment. The whole point of the IRB system is to prevent stuff like that.
What? That's exactly how it works. A bad apple gives off a lot of ethylene which ripens (spoils) the whole bunch.
It is very varied. There are a lot of good and enjoyable stories out there on youtube and podcasts for anyone interested.
With every pentesting engagement I've had, there always were rules of engagement, and what kind of things you are and are not allowed to do. They even depend on what kind of test you are doing. (for example: if you're testing bank software, it matters a lot if you test against their production environment or their testing environment)
Fool me once. Why should they waste their time with extra scrutiny next time? Somebody deliberately misled them, so that's it, banned from the playground. It's just a no-nonsense attitude, without which you'd get nothing done.
If you had a party in your house and some guest you don't know and whom you invited in assuming good faith, turned out to deliberately poop on the rug in your spare guest room while nobody was looking .. next time you have a party, what do you do? Let them in but keep an eye on them? Ask your friends to never let this guest alone? Or just simply to deny entrance, so that you can focus on having fun with people you trust and newcomers who have not shown any malicious intent?
I know what I'd do. Life is too short for BS.
Because well funded malicious actors (government agencies, large corporations, etc) exist and aren't so polite as to use email addresses that conveniently link different individuals from the group together. Such actors don't publicize their results, aren't subject to IRB approval, and their exploits likely don't have such benign end goals.
As far as I'm concerned the University of Minnesota did a public service here by facilitating a mildly sophisticated and ultimately benign attack against the process surrounding an absolutely critical piece of software. We ought to have more such unannounced penetration tests.
> I sent patches on the hopes to get feedback. We are not experts in the Linux kernel and repeatedly making these statements is disgusting to hear.
this is after they're caught, why continue lying instead of apologizing and explain? Is the lying also part of the experiments?
On top of that, they played cards, you can see why people would be triggered by this level of dishonesty:
> I will not be sending any more patches due to the attitude that is not only unwelcome but also intimidating to newbies
Yes, malicious actors have a head start, because they don't care about the rules. It doesn't mean that we should all kick the rules, and compete with malicious actors on this race to the bottom.
This "attack" did not reveal anything interesting. It's not like any of this was unknown. Of course you can get backdoors in if you try hard enough. That does not surprise anybody.
Imagine somebody goes with an axe, breaks your garage door, poops on your Harley, leaves, and then calls you and tells you "Oh, btw, it was me. I did you a service by facilitating a mildly sophisticated and ultimately benign attack against the process surrounding an absolutely critical piece of your property. Thank me later." And then they expect you to get let in when you have a party.
It doesn't work that way. Of course the garage door can be broken with an axe. You don't need a "mildly sophisticated attack" to illustrate that while wasting everybody's time.
"It was my brother on my unsecured computer" is an excuse I've heard a few times by people trying to shirk responsibility for their ban-worthy actions.
Geographic proximity to bad actors is sometimes enough to get caught in the crossfire. While it might be unfair, it might also be seen as holding a community and it's leadership responsible for failing to hold members of their community responsible and in check with their actions. And, fair or not, it might also be seen as a pragmatic option in the face of limited moderation tools and time. If you have a magic wand to ban only the bad-faith contributions by the students influenced by the professor in question, I imagine the kernel devs will be more than happy to put it to use.
Is it really just the one professor, though?
Basically, yes. The kernel review process does not catch 100% of intentionally introduced security flaws. It isn't perfect, and I don't think anyone is claiming that it is perfect. Whenever there's an indication that a group has been intentionally introducing security flaws, it is just common sense to go back and put a higher bar on reviewing it for security.
Whether or not this indicates flaws in the review process is a separate issue, but I don't know how you can justify not reverting all the commits. It'd be highly irresponsible to leave them in.
What I strongly disapprove of the researcher is that apparently no steps are taken to prevent real world consequences of malicious patches getting into kernel, I think the researcher should:
- Notify the kernel community promptly once malicious patches got past all review processes.
- Time these actions well such that malicious patches won't not get into a stable branch before they could be reverted.
----------------
Edit: reading the paper provided above, it seems that they did do both actions above. From the paper:
> Ensuring the safety of the experiment. In the experiment, we aim to demonstrate the practicality of stealthily introducing vulnerabilities through hypocrite commits. Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF bugs will not be merged into the actual Linux code. In addition to the minor patches that introduce UAF conditions, we also prepare the correct patches for fixing the minor issues. We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our correct patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. All the UAF-introducing patches stayed only in the email exchanges, without even becoming a Git commit in Linux branches. Therefore, we ensured that none of our introduced UAF bugs was ever merged into any branch of the Linux kernel, and none of the Linux users would be affected.
So, unless the kernel maintenance team have another side of the story. The questions of ethics could only go as far as "wasting kernel community's time" rather than creating real world loop holes.
This time two reviewers noticed that the patch was useless, and then Greg stepped in (three weeks later) saying that this was a repetition of the same bad behavior from the first study. This got a response from the author of the patch, who said that this and other statements were “wild accusations that are bordering on slander”.
https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
I'd hate to be the PhD student that wastes away half a dozen years of his/her life writing a document on how to sneak buggy code through a code review.
More than being pointless and boring, it's a total CV black hole. It's the worst of both worlds: zero professional experience to show for, and zero academic portfolio to show for.
Just because their actions didn’t cause damage doesn’t mean they weren’t negligent.
The limits of code review are quite well known, so it appears very questionable what scientific knowledge is actually gained here. (Indeed, especially because of the known limits, you could very likely show them without misleading people, because even people knowing to be suspicious are likely to miss problems, if you really wanted to run a formal study on some specific aspect. You could also study the history of in-the-wild bugs to learn about the review process)
That's factually incorrect. The arguments over what constitutes proper code reviews continues to this day with few comprehensive studies about syntax, much less code reviews - not "do you have them" or "how many people" but methodology.
> it appears very questionable what scientific knowledge is actually gained here
The knowledge isn't from the study existing, but the analysis of the data collected.
I'm not even sure why people are upset at this, since it's a very modern approach to investigating how many projects are structured to this day. This was a daring and practical effort.
Under that logic, it's ok for me to run a pen test against your computers, right? ...because I'm only wasting your time.... Or maybe to hack your bank account, but return the money before you notice.
Slippery slope, my friend.
> Under that logic, it's ok for me to run a pen test against your computers, right?
I think the standard for an individual user should be different than that for the organization who is, in the end, responsible for the security of millions of those individual users. One annoys one person, one prevents millions from being annoyed.
Donate to your open source projects!
It would give the University some notoriety to be able to claim "We introduced vulnerabilities in Linux". It would put them in good terms with possible propietary software sponsors, and the military.
I don't think this necessarily follows. Rather it is fundamentally a resource allocation issue.
The kernel team obviously doesn't have sufficient resources to conclusively verify that every patch is bug-free, particularly if the bugs are intentionally obfuscated. Instead it's a more nebulous standard of "reasonable assurance", where "reasonable" is a variable function of what must be sacrificed to perform a more thorough review, how critical the patch appears at first impression, and information relating to provenance of the patch.
By assimilating new information about the provenance of the patch (that it's coming from a group of people known to add obfuscated bugs), that standard rises, as it should.
Alternatively stated, there is some desired probability that an approved patch is bug-free (or at least free of any bugs that would threaten security). Presumably, the review process applied to a patch from an anonymous source (meaning the process you are implying suffers from a lack of confidence) is sufficient such that the Bayesian prior for a hypothetical "average anonymous" reviewed patch reaches the desired probability. But the provenance raises the likelihood that the source is malicious, which drops the probability such that the typical review for an untrusted source is not sufficient, and so a "proper review" is warranted.
> it means that an unknown attacker deliberately concerting complex kernel loop hole would have a even higher chance got patches in.
That's hard to argue with, and ironically the point of the research at issue. It does imply that there's a need for some kind of "trust network" or interpersonal vetting to take the load off of code review.
Nobody can assure that.
But realistically, when you find out a submitter had malicious intent, I think it's 100% correct to revisit any and all associated submissions since it's quite a different thing to inspect code for correctness, style, etc. as you would in a typical code review process versus trying to find some intentionally obfuscated security hole.
And, frankly, who has time to pick the good from the bad in a case like this? I don't think it's an overreaction at all. IMO, it's a simplification to assume that all associated contributions may be tainted.
Linux is set up to benefit the linux development community. If UMinn has basically no positive contributions, a bunch of neutral ones and some negative ones banning seems the right call.
Its not about fairness, its about if the hurts outweigh the benefits.
I think the best way to make this expectation reality is putting in the work. The second best way is paying. Doing neither and holding the expectation is a way to exist certainly, but has no impact on the outcome.
The reviews were done by kernel developers who assumed good faith. That assumption has been proven false. It makes sense to review the patches again.
Given that some patches may have made it though with holes, you pull them and re-approach them with a different mindset.
> it's the linux kernel. Think about what it's powering and how much risk there is involved with these patches
Perhaps the mindset needs to change regarding security? Actual malicious actors seem unlikely to announce themselves for you.
Specifically, I think the three malicious patches described in the paper are:
- UAF case 1, Fig. 11 => crypto: cavium/nitrox: add an error message to explain the failure of pci_request_mem_regions, https://lore.kernel.org/lkml/20200821031209.21279-1-acostag.... The day after this patch was merged into a driver tree, the author suggested calling dev_err() before pci_disable_device(), which presumably was their attempt at maintainer notification; however, the code as merged doesn't actually appear to constitute a vulnerability because pci_disable_device() doesn't appear to free the struct pci_dev.
- UAF case 2, Fig. 9 => tty/vt: fix a memory leak in con_insert_unipair, https://lore.kernel.org/lkml/20200809221453.10235-1-jameslou... This patch was not accepted.
- UAF case 3, Fig. 10 => rapidio: fix get device imbalance on error, https://lore.kernel.org/lkml/20200821034458.22472-1-acostag.... Same author as case 1. This patch was not accepted.
This is not to say that open-source security is not a concern, but IMO the paper is deliberately misleading in an attempt to overstate its contributions.
edit: wording tweak for clarity
..."Because if we're lucky tomorrow, we won't have to deal with questions like yours ever again." --Firesign Theater, "I Think We're All Bozos on the Bus"
Yet we do nothing about it? I wouldn't call that jaw-droppingly obvious, if anything, without this, I'm pretty sure that anyone would argue that it would be caught way before making it way into stable.
What they do almost universally lack is enough people making positive contributions (in time, money, or both).
This "research" falls squarely into the former category and burns resources that could have been spent on the latter.
Welcome to academia. Where a large number of students are doing it just for the credentials
Immigrant graduate students with uncertain future if they fail? Check.
Vulnerable students whose livelihood is at mercy of their advisor? Check.
Advisor whose career depends on a large number of publication bullet points in their CV? Check.
Students who cheat their way through to publish? Duh.
Question for legal experts,
Hypothetically if these patches were accepted and was exploited in the wild; If one could prove that they were exploited due to the vulnerabilities caused by these patches can the University/ Prof. be sued for damages and won in an U.S. court (or) Would they get away under Education/Research/Academia cover if any?
To me, seems to indicate that nation state supported evil hacker org (maybe posing as an individual) could place their own exploits in the kernel. Let's say they contribute 99.9% useful code, solve real problems, build trust over some years, and only rarely write an evil hard to notice exploit bug. And then, everyone thinks that obviously it was just an ordinary bug.
Maybe they can pose as 10 different people, in case some of them gets banned.
See: https://www.reuters.com/article/us-usa-security-siliconvalle...
"As U.S. intelligence agencies accelerate efforts to acquire new technology and fund research on cybersecurity, they have invested in start-up companies, encouraged firms to put more military and intelligence veterans on company boards, and nurtured a broad network of personal relationships with top technology executives."
Foreign countries do the same thing. There are numerous public accounts of Chinese nationals or folks with vulnerable family in China engaging in espionage.
It's difficult to protect against trusted parties whom you assume, with good reason, and good-faith actors.
A perfectly security system is only realized by a perfectly inefficient development process.
We can get better at lessening the efficiency tax of a given security level (through tooling, tests, audits, etc), but for a given state of tooling, there's still a trade-off.
Different release trains seem the sanest solution to this problem.
If you want bleeding-edge, you're going to pull in less-tested (and also less-audited) code. If you want maximum security, you're going to have to deal with 4.4.
How to solve this "issue" without putting too much process around it? That's the challenge.
Sarcasm aside, pentesting/redteaming is only ethical if the target consents to it! Please don't try to prove your point the way these researchers have.
If the researcher has sent these patches under a different identity, that would be just like how malice contributions appear. The maintainers won't assume malice, waste a bunch of time communicating with the bad actor, and may NOT revert their previous potentially harmful contribution.
And as for the solutions, their contribution is nil. No suggestions that haven't been suggested, tried and done or rejected a thousand times over.
What? Are you actually trying to argue that "researchers" proved that code reviews don't have a 100% success rate in picking up bugs and errors?
Specially when code is pushed in bad faith?
I mean, think about that for a minute. There are official competitive events to sneak malicious code that are already decades old and going strong[1]. Sneaking vulnerabilities through code reviews is a competitive sport. Are we supposed to feign surprise now?
* a black hat writes malware that proves to be capable of taking out a nation's electrical grid. We know that such malware is feasible.
* a group of teenagers is observed to drop heavy stones from a bridge onto a motorway.
* another teenager pointing a relatively powerful laser at the cockpit of a passenger jet which is about to land at night.
* an organic chemist is demonstrating that you can poison 100,000 people by throwing certain chemicals into a drinking water reservoir.
* a secret service subverting software of a big industrial automation company in order to destroy uranium enrichment plants in another country.
* somebody hacking a car's control software in order to kill its driver
What are the security implications of this? That more money should be spent on security? That we should stop to drive on motorways? That we should spent more money on war gear? Are you aware how vulnerable all modern infrastructure is?
And would demonstrating that any of these can practically be done be worth an academic paper? Aren't several of these really a kind of military research?
The Linux kernel community does spend a lot of effort on security and correctness of the kernel. They have a policy of maximum transparency which is good, and known to enhance security. But their project is neither a lab in order to experiment with humans, nor a computer war game. I guess if companies want to have even more security, for running things like nuclear power plants or trains on Linux, they should pay for the (legally required) audits by experts.
As per the attack surface described in the paper (section IV). Because (III, the acceptance process) is a manpower issue.
Have they submitted patches to any projects other than the kernel?
"... planning on recording the event to show it on YouTube for ad revenue and Internet fame."
In this case, the offender's friends are benefiting from the research. I think that needs to be made important. The university benefits from this paper being published, or at least expected to. That should not be overlooked.
If you want to talk about a state level actor, I hate to break it to you, but they have significantly more powerful and stealthier 0-day exploits that are a lot easier to exploit than a tactic like this. Guess what's the last thing you want to have happen when you commit cybercrime? Do it in public with where there's an immutable record that can be traced back to you, and cause a giant public hubbub, maybe? So, I can't imagine how someone could think there's anything noteworthy about this unless they were unaware of that.
That's somewhat the unintentional humor and irony of this situation -- all the researchers accomplished was proving that they were not just unethical but incompetent.
However, I don't agree that what happened was abuse or that it should be deterred. Responding in a hostile manner to an isolated demonstration of a vulnerability isn't constructive. People rightfully get angry when large companies try to bully security researchers.
You question if this vulnerability is worth worrying about due to the logistics of exploiting it in practice. Regardless of whether it's worth the effort to exploit I'd still rather it wasn't there (that goes for all vulnerabilities).
I think it would be much easier to exploit than you're letting on though. Modern attacks frequently chain many subtle bugs together. Being able to land a single, seemingly inconsequential bug in a key location could enable an otherwise impossible approach.
It seems unlikely to me that the immutable record you mention would ever lead back to a competent entity that didn't want to be found. There's no need for anything more than an ephemeral identity that successfully lands one or two patches and then disappears. The patches also seem unlikely to draw suspicion in the first place, even after the exploit itself becomes known.
In fact it occurs to me that a skilled and amoral developer could likely land multiple patches with strategic (and very subtle) bugs from different identities. These could then be "discovered" and sold on the black market. I see no convincing reason to discount the possibility of this already being a common occurrence.
The only sensible response I can think of is a focus on static analysis coupled with CTF challenges to beat those analysis methods.
They could discuss the idea and then perform the test months later? With the amount of patches that had to be reverted as precaution the test would have been well hidden in the usual workload even if the maintainers knew that someone at some point in the past mentioned the possibility of a pen test. How long can the average human stay vigilant if you tell them they will be robbed some day this year?
As to the backdated review now being undertaken, as far as I'm concerned that decision is squarely on the maintainers. (Honestly it comes across as an emotional outburst to me.)
It seems to qualify per §46.102(e)(1)(i) ("Human subject means a living individual about whom an investigator [..] conducting research: (i) Obtains information [...] through [...] interaction with the individual, and uses, studies, or analyzes the information [...]")
I don't think it'd qualify for any of the exemptions in 46.104(d): 1 requires an educational setting, 2 requires standard tests, 3 requires pre-consent and interactions must be "benign", 4 is only about the use of PII with no interactions, 5 is only about public programs, 6 is only about food, 7 is about storing PII and not applicable and 8 requires "broad" pre-consent and documentation of a waiver.
It's not worth arguing about this; if you care, you can try to change the law. In the meantime, IRBs will do what IRBs do.
Since IRBs exist to minimize liability, it seems like that would be that fastest route towards change (assuming you have legal standing )
I had a look at section §46.104 https://www.hhs.gov/ohrp/regulations-and-policy/regulations/... since it mentioned exemptions, and at (d) (3) inside that. It still doesn't apply: there's no agreement to participate, it's not benign, it's not anonymous.
IRBs are like the TSA. Imposing annoyance and red tape on the honest vast-majority while failing to actually filter the 0.0001% of things they ostensibly exist to filter.
I'm guessing it passed for similar reasoning, along with the reviewers being unfamiliar with how "vulnerabilities are injected." To get the bad code in, the researcher needed to have the code reviewed by a human.
So if you rephrase "inject vulnerability" as "sneak my way past a human checkpoint", you might have a better idea of what they were actually doing, and might be better equipped to judge its ethical merit -- and if it qualifies as research on human subjects.
To my thinking, it is quite clearly human experimentation, even if the subject is the process rather than a human individual. Ultimately, the process must be performed by a human, and it doesn't make sense to me that you would distinguish between the two.
And the maintainers themselves express feeling that they were the subject of the research, so there's that.
What makes people revieing linux kernel more 'human' than any of the above?
Also US code 1030(a)5 A does not care about software license. Any intentional vulnerability added to code counts. Federal cybercrime laws are not known for being terribly understanding…
Given the completely unavoidable limitations of the review and bug testing process, a maintainer has to react very differently when they have determined that a patch is malicious - all previous patches past from that same source (person or even organization) have to be either re-reviewed at a much higher standard or reverted indiscriminately; and any future patches have to be rejected outright.
This puts a heavy burden on a maintainer, so intentionally creating this type of burden is a malicious action regardless of intent. Especially given that the intent was useless in the first place - everyone knows that patches can introduce vulnerabilities, either maliciously or by accident.
The vast majority of drunk drivers never kill anyone.
> Sending a malicious patch (one that is known to introduce a vulnerability) is a malicious action.
I disagree that it's malicious in this context, but that's irrelevant really. If the patch gets through, then that proves one of the most critical pieces of software could relatively easily be infiltrated by a malicious actor, which means the review process is broken. That's what we're trying to figure out here, and there's no better way to do it than replicate the same conditions under which such patches would ordinarily be reviewed.
> Especially given that the intent was useless in the first place - everyone knows that patches can introduce vulnerabilities, either maliciously or by accident.
Yes, everyone knows that patches can introduce vulnerabilities if they are not found. We want to know whether they are found! If they are not found, we need to figure out how they slipped by and how to prevent that from happening in the future.
Or perhaps it really is a second attempt by his advisor at an evil plot to sneak more buggy patches into the kernel for research purposes? Either way, the response by the maintainers seems rather disproportionate to me. And either way, I'm ultimately grateful for the (apparently unwanted?) attention being drawn to the (apparent lack of) security surrounding the Linux kernel patch review process.
Who then replies with a request for "cease and desist"? Not sure that's the right move for a humble newbie.
I also don't view unannounced penetration testing of an open source project as immoral, provided it doesn't consume an inordinate amount of resources or actually result in any breakage (ie it's absolutely essential that such attempts not result in defects making it into production).
When the Matrix servers were (repeatedly) breached and the details published, I viewed it as a Good Thing. Similarly, I view non-consensual and unannounced penetration testing of the Linux kernel as a Good Thing given how widely deployed it is. Frankly I don't care about the sensibilities of you or anyone else - at the end of the day I want my devices to be secure and at this point they are all running Linux.
That you care about something or not also seems to be irrelevant, unless you are part of either the research or the kernel maintainers. It’s not about your or my emotional inclination.
Acquiring consent before experimenting in human subject is an ethical requirement for research, regardless of whether is a hurdle for the researchers. There is a reason that IRB exists.
Not to mention that they literally proved nothing, other than that vulnerable patches can be merged into the kernel. But did anybody that such a threat is impossible anyway? The kernel has vulnerabilities and it will continue to have them. We already knew that.
So what other things do you think appropriate to not engage in acquiring consent to do based on some perceived justification of ubiquity? It's a slippery slope all the way down, and there is a reason for all the ceremony and hoopla involved in this type of thing. If you cannot demonstrate mastery of doing research on human subjects and processes the right way, and show you've done your footwork to consider the impact of not doing it that way (i.e. IRB fully engaged, you've gone out of your way to make sure they understand, and at least reached out to one person in the group under test to give a surreptitious heads up (like Linus)), you have no business playing it fast and loose, and you absolutely deserve censure.
No points awarded for half-assing. Asking forgiveness may oft times be easier than asking permission, but in many areas, the impact to doing so goes far beyond mere inconvenience to the researcher in the costs it can extract.
>at the end of the day I want my devices to be secure and at this point they are all running Linux.
That is orthogonal to the outcome of the research that was being done, as by definition running Linux would include running with a new vulnerability injected. What you really want is to know your device is doing what you want it to, and none of what you don't. Screwing with kernel developers does precious little to accomplish that. Same logic applies with any other type of bug injection or intentioned software breakage.
In the same way there is no law requiring Linux kernel maintainers to review patches send by this university.
"it was not literally illegal" is not a good reasoning for why someone should not be banned.
Sometimes, complex situations don't have simple analogies. I'm not even sure mine is 100% correct.
Just like bumping into somebody on the roof is normal, but you should always be aware that there’s a chance they might try to throw you off. A researcher highlighting this fact by doing it isn’t helping, even if they mitigate their damage.
A much better way to show what they are attempting to is to review historic commits and try to find places where malicious code slipped through, and how the community responded. Or to solicit experimenters to follow normal processes on a fake code base for a few weeks.
Strangers submitting patches might be completely normal.
Malicious strangers trying to sneak vulnerabilities by submitting malicious patches devised to exploit the code review process is not normal. At all.
There are far more news reports of deranged people throwing strangers under traffic, subways, and trains, than there are reports of malicious actors trying to sneak vulnerable patches.
How could you possibly know that? In fact, I would suggest that you are completely and obviously wrong. Government intelligence agencies exist (among other things) and presumably engage in such behavior constantly. The reward for succeeding is far too high to assume that no one is trying.
But the linux kernel is NOT a security product - it is a kernel. It can be used in entirely disconnected devices that couldn't care less about security, as well as in highly secure infrastructure that powers the world. The ultimate responsibility of delivering a secure product based on Linux lies with the people delivering a secure product based on the kernel. The kernel is a essentially library, not a product. If someone is assuming they can build a secure product by trusting Linux to be "secure" than they are simply wrong, and no amount of change in the Linux dev process will fix their assumption.
Of course, you want the kernel to be as secure as possible, but you also want many other things from the kernel as well (it should be featureful, it should be backwards compatible with userspace, it should run on as many architectures as needed, it should be fast and efficient, it should be easy to read and maintain etc).
That is a complete misunderstanding of the Linux dev process. No one expects the first reviewer of a patch (the person that the researchers were experimenting on) to catch any bug. The dev process has many safeguards - several reviewers, testing, static analysis tools, security research, distribution testing, beta testers, early adopters - that are expected to catch bugs in the kernel at various stages.
Trying to deceive early reviewers into accepting malicious patches for research purposes is both useless research and hurtful to the developers.
Bug bounties are more than a different beast: they are a strawman.
Sneaking vulnerabilities through a code review is even a competitive sport, and it has zero to do with bug bounties.
It's just f** brilliant! :)
And also, if I had to pick between a somewhat inclusive mode of work where some rando can get code included at the slightly increased risk of including malicious code, and a tightly knit cabal of developers mistrusting all outsiders per default: I would pick the more open community.
If you want more paranoia, go with OpenBSD. But even there some rando can get code submitted at times.
I mean, it is no surprise. It is even worse with proprietary software, because you are much less likely to be aware of your own college/employee.
Hell, seeing that the actual impact is overblown in the paper, I think it is a really great percentage caught to be honest, assuming good faith from the contributor.
Frankly universities and academics need to be taken to court far more often. Our society routinely turns a blind eye to all sorts of fraudulent and unethical practices inside academia and it has to stop.
Nor is breaking my computer monitor.
Theft was meant as an example of a type of harm, not a complete list of all types of harm.
Something doesn’t have to be illegal to be harmful.
Sure, but I'm still going to be pretty annoyed with you. And if you've wasted my time by messing with a system or process under my control then I'm probably going to block you from that system or process.
As a really prosaic example, I've blocked dozens - if not hundreds - of recruiter email addresses on my work email account.
I too thought like this till yesterday. Then someone made me realize thats not how getting consent works in these situations. You take consent from higher up the chain, not the people doing the work. So Greg Kroah-Hartmancould could have been consulted, as he would not be personally reviewing this stuff. This would also give you a chance to understand how the release process works. You also have an advantage over the bad actors because they would be studying the process from outside.
But I would like to put in a disclaimer that before getting to that point they could have done so many other things. Review the publicly available review processes, see how security bugs get introduced by accident and see if that can be easily done by a bad actor, etc.
Yes, and if you do it without a heads-up as well that makes you a bad actor. This university is a disgrace and that's what the problem is and should remain.
To take a more realistic example, we could quickly learn a lot more than today about language acquisition if we could separate a few children from any human contact to study how they learn from controlled stimuli. Still, we don't do this research and look for much more complicated and lossy, but more humane, methods to study the same.
I also consider Greg’s response just as much a test of UMN’s internal processes as the researcher’s attempt at testing kernel development processes. Hopefully there will be lessons learned on both sides and this benign incident makes the world better. Nobody was hurt here.
> Nobody was hurt here.
This is where you got me, because while it's clear to me that short-term damage has been done, in the long term I believe you are correct. I believe this event has made the world a safer place.
The purpose of the research was probably to show how easy it is to manipulate the Linux kernel in bad faith. And they did it. What are they gonna do about it besides banning the university?
If a corporation relies upon open sourced code that has historically been written by unpaid developers, if I was that corportion, I would start paying people to vet that code.
https://www.usenix.org/conference/usenixsecurity19/speaker-o...
Edit: Oh now I get it you clever person you. Only took an hour ha.
It's good to call out bad incentive structures, but by feigning surprise you're implying that we shouldn't imagine a world where people behave morally when faced with an incentive/temptation.
I don't think it's fair to say "by feigning surprise you're implying..." That seems to be putting words in GP's mouth. Specifically, they didn't say that we shouldn't imagine a better world. They were only describing one unfortunate aspect of today's academic world.
Here is a personal example of feigned surprise. In November 2012 I spent a week at the Google DC office getting my election results map ready for the US general election. A few Google engineers wandered by to help fix last-minute bugs.
Google's coding standards for most languages including JavaScript (and even Python!) mandate two-space indents. This map was sponsored by Google and featured on their site, but it was my open source project and I followed my own standards.
One young engineer was not pleased when he found out about this. He took a long slow look at my name badge, sighed, and looked me in the eye: "Michael... Geary... ... You... use... TABS?"
That's feigned surprise.
(Coda: I told him I was grateful for his assistance, and to feel free to indent his code changes any way he wanted. We got along fine after that, and he ended up making some nice contributions.)
We'd also have to agree on what "behave morally" means, and this is impossible even at the most basic level.
The main culprit seems to be only Quishu Wu. He is also the one who wrote the paper.