Please stop flooding our projects with AI slop to furnish your CV(neilalexander.dev) |
Please stop flooding our projects with AI slop to furnish your CV(neilalexander.dev) |
> The changes were harmless and correct,
> I closed all three PRs without comment.
What is this article? I get real ‘AI slop PRs’ are nightmares - but I take that to mean the wave of AI generated PRs that read convincingly but are in the end bullshit.
But your example is NOT that. The unreasonable counter to your point: don’t make spelling mistakes in the first place? Or switch off PRs if you don’t want things corrected?
If you don’t have a valid point why post rubbish?
AI code review, to my eyes, just reduce the number of bugs, they do not shorten the review time. Otherwise, it means you are compromising on the direction of the project.
The amount of security advisories and PRs opened is getting out of hands. This is no longer just spell check PRs that you can merge in less than 1 minute of review. Often that's going to be supposed perf improvement, attempts at fixing things that are not broken, or submitting completely new features.
These relatively low effort PRs takes a lot of effort to review or even just to filter out.
But in open-source- process is unpaid for sparse- and the gate-keepers usually have excellent reasons to merge changes not back. If they don't - they are forked.
The reason being that while the software generated "solves" your problem, it generates problems for everyone else using the project.
Like demanding a elevator cabin be tailored to your preferences, ignoring all the other buildings using elevator cabins, to which it no longer is compatible for example.
Reading the tea leaves, this initially said NH instead of HN, which might be a typo, but might also be "hacker news" but in the native language. Which would possibly line up with commit messages.
Anyway, low trust internet is here. Enjoy
Automated PR's can have automated responses. You make a human effort, you get a human response.
The weird flip side is that the average good AI contribution is better than the average “I used no AI” contribution. Perhaps it’s because Homebrew has so many declarative guardrails and is so easy for agents to test.
Since AI makes it easier for incompetent people to look smart, AI can help us expose those people for what they are.
I think if OSS maintainers start pushing that financial burden out to potential contributors then there will be a big outcry.
OSS may become pay-to-play.
I didn’t like the second part though where it starts banning people for “bad behavior”.
I get that AI resistance will be a thing in the next few years, but realistically engineers who plan on being around for the next 20-30 years just need to embrace it instead of being sour about it.
Because of machines, things have arguably improved for humanity as a whole. People who seek more elevated/finer products can still turn to handcrafted products.
Mass software will be produced by AI, and it will scale just fine.
That's a very narrow view of the situation. I agree that this side of the coin exists, but there's also the other side where drinkable tap water is becoming a luxury in much of the global North, giant tornadoes and fires are sweeping the Earth, and temperatures are growing out of control.
So, to make lives more comfortable with machines for some millions of privileged people, they triggered the 6th mass extinction. That's the other side of the coin.
I think you failed to notice the ban would be a reaction to recurrent bad behavior, such as ignoring maintainer requests.
What's your solution to handle accounts which spam projects with nonsense PRs?
> get that AI resistance will be a thing in the next few years (...)
This is a very lazy opinion to express. If you pay attention to the post you're replying to, you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers.
You'd have the same problem if some guy decided to spam the project with a constant barrage of small low-effort nonfunctional PRs.
That AI will be given the power to ban people?
Or that AI will be the ones getting banned?
If it's the former, expect an invite from the AI resistance. :)
If it's the latter, well, that can be fixed by spending more tokens and writing better PRs so that the AI moderator doesn't think they're "low-effort, AI-like". I've seen my share of low-effort PRs from humans, too. Low-effort is low-effort, whether it's from humans or AI.
What's your thoughts on them banning bots?
Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time.
I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.
If I get a review at work from a bot that is annoying, misunderstands, etc I just ignore their reviews, or tell the bot to leave me alone. The bot is there to help me, and if it doesn't - that's not my problem. There's also always a human reviewer. For any automated feedback for PR's I think there should be "manual review" requests. If you disagree and the bot can decide that there is real human effort involved, then just let real humans also make the effort to review.
This is the key idea: to spend the human effort from maintainers, where there has been human effort from contributors. The AI should be able to determine the amount of human effort in a contribution. The bot should be an aid to make sure the human maintainer effort is spent only where there is human contributor effort.
That's a super weird way of putting it, which makes me hesitate to agree with you.
buuuut I agree that the LLM bot review experience is usually miserable, as the persona these things use is insufferable. It's a know-it-all only-speaks-in-imperatives thing, where the speaker lacks the actual merit that would grant them the right to that style.
LLMs can be very useful for code review, but the existing product implementations took a bit too many pages out of Kafka's works and are a bit too real in simulating a dysfunctional bureaucratic org.
Actually, if I see someone doing open source like it's a performative career checkbox, that will not be positive signal, and could easily be negative. I understand that people will do what they need to do to get a job, but that pragmatic career checkboxing itself isn't positive. I will have to look for positive signal elsewhere for that person.
The problem is that our industry has gotten very bad at hiring, and now we have everyone playing all sorts of games with it -- rehearsing Leetcode interviews, bad faith open source contributions, feigning enthusiasm, spamming AI-tweaked resumes, outright cheating on interviews -- rather than focusing on doing good work, and being part of a team.
If you're involved in hiring, and you care about effectiveness and culture, consider pushing back against the prevailing dysfunctional big-corporate-cattle-herding practices. Especially if your company has no excuse to be big-corporate dysfunctional, and can't afford to be.
Why not have these PRs counted differently (by the platform), and/or colored differently in the timeline(s) thus made less visible or more clear?
AI is destroying trust in open source and many other areas and I think this will discourage teams from publishing their source code in the future.
On the other hand, personal connections are becoming even more important, which is unfair to the younger generation and people who don't live near tech hubs.
I didn't agree with your comment when I glanced through it, but then further ahead in the discussion I stumbled upon a whole thread of people commenting that PRs generated by AI coding assistants are strealing merit, and even posters talking about the need to tell apart AI PRs from AI-free PRs with proof of work schemes just to feed the merit angle.
Crazy times we live in.
And it is true, but that just means the bar for expectations has been raised thanks to early open source success. Tarball drops don't cut it anymore (unless you're Alphabet, Inc.) and you actually have to foster a community around what you release in order to attract bug reports and contributions. And you need a code of conduct, and a code of conduct enforcement committee, and a steering committee or board of directors, and...
I suspect many people are like me since it feels like a very normie position to take. That means that contributors are even more likely to be useful because both the drive-by genuine contributors have chosen otherwise and these contributors have increased.
It must be quite painful to be an open source maintainer right now. One wonders what to make of such projects in a future where a feature-list is a sufficient prompt.
Claude doesn't read AGENTS.md automatically, it only looks for CLAUDE.md.
Sounds like an opportunity for GitHub/others to better filter these.
Open source contribution on github specifically is no longer about building and maintaining something, its a parasocial signal that has more in common with linkedin hustle bloggers.
Instead opensource is full of cynical projects and real important projects that do amazing work with maintainers who shoulder a great burden having to listen to people bitch about them for features and fixes while giving nothing and dealing with bad PR's (or even just having to evaluate good and mediocre PR's). Often the only pay is that you can go to HR and shit corp with your CV and say "Here is what I have done, can I now be treated like a bitch for more money".
I think opensource has allowed a lot businesses to treat developers with contempt and this is fully manifest with the way they conceive of AI agent's and Open source has probably become a vector for demoralization and devaluation more then anything.
Out of curiosity, I then asked Claude to draft an enhancement request for a new feature. It created a highly detailed issue and went on to generate the corresponding code for the enhancement.
I haven't submitted a pull request because the code is written in a language I don't know. Part of me considered submitting it anyway since it works and could be useful, but after reading this post, I think I'll just keep it under wraps.
Does anyone have any advice when to decline CVEs?
I got PRs that either never worked or even broke stuff
In my company we finally have it all documented, it’s super easy to find which PR made the regression and our internal policies and rules clearly state that you are the owner of the PR, not the AI so the ownership remains on human side, AI mistake is your mistake so everyone pays extra attention to that.
So do you have the project’s best interest at heart or not? If you’re more concerned about the intent of a valid contribution than the content, why don’t you ask yourself where your intent is? You rejected a valid contribution based on unverified vibes about the person’s intent, instead of assuming they were just being helpful.
PRs submission should be closed, and only issue submission, with a prompt to reccreate and fix the issue should be shared. This way the prompts users are using to do things - the real contribution - will make their way to maintainers who can then choose to convert it into a PR and merge.
Same story as SEO, I presume. Google surely understood their ranking criteria deeply, yet couldn't exactly win that fight.
There weren't that many with high ranking. I could filter out most of them in like a week of working normally and just right clicking them with the blacklist extension in Firefox.
Nowadays it is probably harder though when Google have broken the internet culture.
My big hope for LLM coding is that it will finally force us to focus on minimizing the amount of code that is out there instead of ever producing more.
Obviously the best way to use AI to furnish your CV is to build your own project using AI.
What it does well is coordination between corps, working in public (while on a corp payroll) and extracting some drive-by-PR value out of random third-parties.
The closer your project is to that shape, the better it works for you. The further it is away, the more pain you will feel.
This has nothing to do with AI slop. AI slop just turned up a few knobs that were already there.
One big problem I have with them is that they take away time from project maintainers for reviewing and helping to get the PRs into shape, which then can't be spent on other, more important things. I feel like the "good first issue" GitHub label is specifically attracting these kinds of contributions.
It's not a black-or-white thing though, and you need to tell apart folks who produce slop PRs against any arbitrary repo, from folks using AI to contribute in a sensible way. We've tried to codify some rules in our contribution guide [1]:
- PRs from apparent bot accounts are closed - PRs from users who file large numbers against random repos are closed - You're welcome to use AI, but you need to stand behind your PR and be able to explain it
I'm sure we'll adjust those rules over time, but since we have instantiated them, it definitely has become easier to deal with AI PRs and handle them in an a relatively objective way. It absolutely means that sometimes a PR will be closed which could have been an improvement, but I think this is the right thing to do given the circumstances.
[1] https://github.com/hardwood-hq/hardwood/blob/main/CONTRIBUTI...
After reading your comment, I had a thought that the key could be to separate random, drive-by open source contributions (which are easily gamed by pointing a bunch of AI agents at GitHub) from committing to maintain a small number of open source/community projects for the long term.
Helicopter parents and then young adults are already long-term gaming metrics, before they start high school, just to get into college, and it seems that the same kinds of thinking persist through first job and rest of career.
Being seen as a long-term maintainer of an open source project is now trivial, compared to some of the other things people already do.
Offhand, ideas:
1. Use criteria the metrics-gamers won't think of, and don't tell them what it is. (Why this might work: the metrics-gaming mindset might blind people to thinking outside that mindset. Why this might not work: you need someone not blinded to apply the criteria.)
2. Do things that are unattractive to the metrics-gamers, but attractive to people who aren't. (Though you might still get overwhelmed by near-zero-cost AI spamming applications anyway.)
3. Accept that too many current workers were brought up with metrics-gaming mindset, and the others are too hard to find in the noise, and select and align metrics-gamers in game-theoretic ways. (Extreme example: you pay minimum wage, and there are no promotions nor titles nor performance evaluation metrics not anything except success of the company, but everyone gets real equity that's worth zero unless the team together delivers and wins, in which case everyone gets equally wealthy. The trick would be convincing the dumber metrics-gamers that they can't just nod and game this, and if they don't play like a team member for the team to win, they will be voted off the island and get zero, and possibly left in a ditch.)
Reminds me of this xkcd: https://www.explainxkcd.com/wiki/index.php/2899:_Goodhart%27...
Unless the change is an obvious improvement, it has to be worth the time for the maintainers to spend attention reviewing it (and supporting the code forever, and all the rest).
Even if these particular changes are "harmless" and easy to review, accepting them sets a precedent that encourages an unsustainable flood of AI-generated changes that will overwhelm the project.
About setting a precedent: if you think like that you probably never accept any contributions at all, by humans or otherwise. Contributions by strangers have always been very minor things in general, things that the author cared enough about to make a pull request. In this case I don’t see the difference except for the fact that ai was used. If you just don’t accept ai at all , fine. But just be clear about it. In this case they are calling spelling correction slop. That’s not what slop means. Pretending human contributions were always “great” and nothing else would do is kind of ridiculous and shows a lack of experience in open source.
True, it is actually negative sum, because fake merit erodes trust in real merit, as it makes it a lot harder to spot the latter.
How do you distinguish those who know what they are doing from those who only can write a prompt and copy the result?
Making the haystack bigger is a bad idea if you search the needle
Linux lately received a lot of fixes for drivers nobody has cared about for ages. They decided to remove them. Maybe they should have done this earlier, I don't know, but that happened because every code change comes with cost down the line. For real contributers, this cost is much more limited and they have a higher probability of burden the cost down the line.
Pure AI MRs are just rude, because it's just "here take this, don't really care what this is, but AI said it being good, I won't be around, so you will have to deal with this code after the merge and you better don't overlook anything, because I personally don't really know what I am doing, I only know how to have AI making something that looks convincing. Good luck fixing the bugs this change introduces".
On the other hand: lets be careful not to dismiss valid but inexperienced attempts to contribute. Having someone want to genuinely contribute to your software is incredibly generous. If someone seems like they're trying its better to give advice than act like a snob because its not good enough. Often pull requests only need small fixes to get in, anyway.
maybe, the maintainer would need to review them to determine that, and that can be time-consuming especially because LLMs tend to provide walls of sometimes dense text
since the human submitting the PR (if there is one) couldn't be bothered to understand whether the change is a valid one (and therefore submit the change as their own work), why should the maintainer take on that work. if they wanted an LLM to find bugs in their code they could run it themselves with less work than sorting through LLM-generated PRs.
I don't think that there is a technical solution to be found here, as the problem is anything but technical.
__
A hack/trap:
Comment "Ah yes thanks a lot for the hint :)", then make the changes yourself.
Then see how the person reacts to that.
Hack the grifters. Hack the planet.
- code gets re-regenerated over and over again, so waste of resources
- no signal back upstream that it is indeed a genuine need that is in fact so important some users did dedicate resources for it, maybe more users would benefit from it
By the way, interesting person to have 512 IPv4s (and two /44s? Aren't those enormous?). Root-level http intentionally user/pass gated on your site? I was curious to see.
You just scratch your own itch and you're done. There is no need to "attract contribution", usually, as usually, your project will do exactly what you need.
where is sqlite's GitHub? It doesn't have one because it drops tarballs.
Yes that's weird sorry. What i meant is it belongs in the uncanny valley. I don't have any good/bad feelings about test suites and CI feedback, it's very binary. Receiving a mail about a comment on my PR raises my hopes that it reached the maintainers (or interest from other contributors); receiving LLM text destroys my expectations.
Then, about the content itself, it has a very patronizing tone. Saying out loud every single thing that's in my +25/-3 PR is really infantilizing. I would probably have slightly less negative reactions if it only commented to suggest improvements, instead of patting me on the back. See this example: https://github.com/TwiN/gatus/pull/1712#pullrequestreview-46...
Change is bad. It takes engineers time to understand and approve. It confuses users. Changes should be made for good reasons, not frivolously.
Just because writing code is easier does not mean you should write more of it. It is more important than ever to carefully consider what you work on and what work you accept.
The user says "this is just spelling corrections" but a maintainer - a volunteer - needs to read all the files and make sure it's harmless and doesn't introduce security vulnerabilities, change meanings, and so on. That's not free.
You’re vastly overestimating the value of these “fixes” that now go to private forks, because maintainers also have access to LLMs and can prompt, probably better than a drive by fix since they have better understanding and context of the codebase.
Therefore I believe, in most cases, keeping those fixes to private forks is not a noticeable loss for the OSS projects.
1. https://naturalhistory.si.edu/education/teaching-resources/p...
2. https://www.endangeredspeciesinternational.org/overview5.htm...
But there seemed to be some time lag for a SEO spam sites to climb. Removing the ones that make it would mess up the economics at the time. Like the Stack Overflow mirror sites or like spam phone number pages with no other information but the number.
And I guess plenty of search queries would have no moderation. But I mean I spent like 10 minutes. You could have 10 FTE doing that.
/48 is the smallest accepted IPv6 advertisement and /24 the smallest allowed IPv4 advertisement (on the net at least). IPv6 assignments are done on the nibble (/48, /44, /40, ...). Since I only have two sites I could have gotten away with a /23 and something like a /44 but then the RIR pointed out it'd probably be cleaner to manage to get a /40 and give each site a /44 at the same renewal cost. As to why, I've just always worked in or near the networking field and so it seemed cheaper/easier to run it all myself in the long run.
Through another very long story, the Org ID + ASN + IPv4 space + IPv6 space ended up costing something like ~$1k the first year and then ~$500 in yearly renewals. I'm not legally able to resell these for the massive market rate prices of /24s because of them being leased for $0 specifically to dual stack my core hosted services like DNS with my own public IP, but that's fine as I got them to use :).
The largest assignment I've ever had the pleasure of getting and deploying was a /32 (... of IPv6!) for an extremely large healthcare system. Certainly no great record smasher, but I thought it was cool to work on an IPv6 subnet so big it could have been an IPv4 route nonetheless! That one was still also very sparse of course, but similar kinds of reasoning ended up getting to that being much easier/better to work with than a dense assignment.
I also have a small MAC address range (a classical "OUI" sized range is 24 bits of addresses, I have 12 bits of addresses) for a small run of some hardware I was making for a particular customer use case. With the leftovers I put my laptop on it and get the occasional "how the hell?" when they see Wireshark decode my laptop MAC as "Dan_Smith_xx_xx" while packet capturing to troubleshoot.
What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry.
> you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers.
What I noticed is that this is literally what AI companies encourage. This is intended use of AI.
You should really read the comments you are replying to. The whole point of the proposal is to provide actionable feedback to the people operating these drive-by PR bots, and kick out repeat offenders to prevent them from generating noise.
> What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry.
I think you're trying too hard to come up with imaginary scenarios. Also, keep in mind that the objective is to steer contributions to be aligned with the project goals and help maintainers not waste time with low-quality noise. The goal is not to impose a fundamentalist militant stance against AI.
This plan will kick no bots, because bots dont mind discussion with another bot. However, falsely flagged people will realize they are talking with bot prompted to be obtuse, get annoyed anf then will be banned
> the objective is to steer contributions to be aligned with the project goals
That was not the objective as stated.
If the maintainer wants a LLM to review my PRs, let the maintainer use a LLM. Don't make me deal with abrasive chatbots when you want to deal with abrasive chatbots. I don't want an LLM to pre-screen my work to decide if a human should review it; a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
But that's the problem at hand: if there are 999 low-effort automated AI contributions for every human contribution, either all 1000 contributions must be pre-screened to gauge "human effort" and the human efforts get human review, or all PR's get 1/1000 the human attention.
I believe that that was in part simply people defending their reality.
Because if it's not the maintainers fault, then why did it fail? What are the implications of that? What does it mean for me personally? Do I have to do something differently? Do I have to change my assumptions?
People do not like these feelings. They want to simply push them away. The easiest way of doing so is blaming that guy and being done with the negative emotion.
Most people do suck at this whole "being human" business.
The commit in itself might add some minuscule value to the project, but when you take into account the time it takes to check the commit and approve it, you're probably back at negative value again.
Plus, there's the problem of people doing this in an adversarial way. How can you tell whether a random commit touching 100 files is the work of an enthusiastic beginner, or a malicious hacker that's hidden a subtle attack somewhere? Well, you need to spend your time carefully checking it.
So it's way more pragmatic just to set up some automated filter to put all of this stuff in the bin.
Oh and by the way:
> I wouldn’t think a commit fixing typos ever attributed merit to a developer.
The first sentences of the article in question:
> Successful contributions to open source projects are a kind of currency. GitHub in particular encourages this in a number of ways: [...] Potential hiring managers often take note of this. Recruiters often find and screen candidates this way.
If you judge first time contributors on accepted PRs that's as dumb as doing it based on their activity. It tells you nothing about quality. You're the stupid one for assuming all maintainers are as strict as you are.