SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.

Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.

The "moat" is data. And compute. But mostly data.

It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.

One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.

There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.

(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)

Because it's all "AI" now <insert Ancient Aliens meme>

Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).

Code is there to be read and understood by the human developers who come later.

The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.

Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.

If an LLM generated a 230 file change they are also capable of going back and breaking it up.

One thing they are bad at is comments that are succinct since they almost only ever add words.

My dev branch got broken when I rebased to main after a week of drift.

I had to bisect over around a million commits to the monorepo to find the culprit.

Not quite. Ironically, they get PR-reviewed by someone else's agent. The humans in between are meat-proxies, pressing OK buttons.

"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)

... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.

If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.

In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.