Automating AI Away(replicated.live) |
Automating AI Away(replicated.live) |
BTW, you should probably fix the Beagle link on your homepage: https://replicated.live/beagle/
In other words, are there places where a one liner for the agent would be more reliable than markdown instructions and crossing fingers?
I look at it this way... I wrote scripts over the years to make my life easier. Do the same for your agents and free their attention for the parts that matter.
[1]: https://github.com/gritzko/jab
I also aspire to make one post a day. To be continued.
Formal methods, as in proof of correctness, have been around for decades (I was doing that stuff in the 1980s) but pushing the proofs through was too laborious. The seL4 verification effort reportedly used over a decade of people time.
The idea is that if you have a formal specification of what you want to happen, you can get a LLM to do the struggling with the proof system to get it right. It's a good task for an LLM, because there's feedback from the prover.
I'd like to see more non-trivial examples of this. People keep republishing verifications of greatest common divisor or stack algorithms, which was done decades ago.
LLMS should be abstracted out of a process as soon as practicable, replaced with deterministic processes or procedures. Otherwise you’ve built the world’s most fragile process at the mercy of token cost, vendor hostility, geopolitics, and model deprecation.
Developers and developer-adjacent, technical people tend to think this way on their own... but every business has dark corners where repetitive, manual things still happen. We're leaning a lot on training and even org-wide LLM instructions to try and let the LLM (by its own assessment) be the vehicle use to codify a process and turn it into some good old-fashioned reviewable, deterministic automation.
Just to be clear, software development itself is not deterministic, though? The software developer pushes a given business process from less-deterministic toward more deterministic? When we say we’ve “abstracted LLMs out of a process” we’d also say that we’ve abstracted software developers out that process as well?
And then... yeah. You got it exactly right. Once a problem or process is deterministic, that's the wrong application of an LLM.
But I had never quite thought of it in these exact terms. The way I've been thinking about it up until now is that the very best way to use LLMs is to have them produce tools. The tools get to stay reliable and predictable. They boost your performance. But I think you found the more general abstraction of the same idea. Tool-making is not deterministic. But the tools themselves can be. That's why it fits. Trying to stuff LLMs into what's otherwise a deterministic process is an absurd waste and error-prone.
Smart. I like it.
This sounds made up or your workplace is rather odd to say the least. Maybe english isn't your first language and "threatened" is not the correct word?
I've genuinely never considered it from this angle before.
Coming up with simple specs is not necessarily easy. You could say that is kind of what math is about. That’s how we actually make progress: find those cases where simple specs are possible and build upon them. That’s the kind of library made for eternity.
Thus why we replaced computers (flesh and blood people writing out calculations) with computers (silicon-based number-crunching machines).