56% merge rate on 316 cold OSS PRs in a week(github.com) |
56% merge rate on 316 cold OSS PRs in a week(github.com) |
Spamming OSS with PR created by AI will cause angry replies here. It looks like you are doing an effort to improve them, but IMHO it still looks like spam.
My recommendation would have been to start with a smaller sample, let's say 20 and manually review each one of them and ensure they have a minimum amount of quality and promises to destroy the photo of a kitten for each rejected PR. (Some people would complain anyway.) And then iterate, and try to keep a 50% acceptation rate.
I saw your post yesterday https://news.ycombinator.com/item?id=48112050 and honestly I expected a lower than 20% acceptation rate. There is just too many people sending bad AI PR that I would have consider a 10% high bar. [Edit: To clarify, 20% is much more than what I expected yesterday.]
IIUC they are not "cold unrequested" PR (I don't know the correct name. You are sending PR that fix an "issue" in the github repo, so I guess it's a better starting point that most people sending bad AI PR.
So far, I got the harness to create 368 PRs, 110 of which were resolved. Of those, I got a 56% merge rate.
The harness: https://github.com/kimjune01/sweep
But as soon as I realized what I was doing to maintainers, I worked all night to build the same pipe but backwards for PR review.
The defense: https://github.com/kimjune01/immune
Which means: bots on the contributor side, bots on the maintainer side. The surprise? What's left is two humans on either end. I accidentally invented a H2H protocol.
I'm happy to share what I learned.