Why harnesses matter for energy (and token) efficiency(neuralwatt.com) |
Why harnesses matter for energy (and token) efficiency(neuralwatt.com) |
Another aspect to this is model training, since bots are now being trained with a harness, and that has the tools it gets most burned in to weights. In my own harness, I tried Read,Edit,Write tools which always put the latest text of a file upfront, only one copy, to reduce context window growth as the bot tended to reread a file before/after edits. While this greatly slowed window growth, cache hits dropped to ~50%. If you give the bot a Grep and Read with Offset/Limit, this aligns with training and reduces window growth, even with the read-before-write pattern. You can see the pattern of grep-read when it does the default "use bashisms for everything instead of the dedicated tools"
I'm now exploring ways to remove the need for a shell tool completely, for various reasons, but notably limiting potential for loss of control. Early experiments hint this will not work well due to the heavy shell usage in training, but I'm not throwing in the towel just yet!
One level deeper is all numbers though.
And eventually it's all 0 and 1s!
/s
All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.
Getting 99% cache hit rates while providing a good user experience is the hard part!
In my opinion, Harness has nothing to do with energy (ie cost). They all use the same conversational requests so the efficiency is mostly in the prompt (instruction and data) as well as the chosen model.
Choose wisely.
For efficiency, providers should cache the initial prefix (ie system prompt and tools) of harnesses, not sure if they do all that.
The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.
Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.
Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.
Happy to discuss the methodology.
What would be helpful would be a cheap API call that would give me the energy cost estimates:
> Given some prefix-cache uuid, what is the cost (in mW of GPU energy) of adding X tokens that will branch off from what's been cached at offset Y tokens.
Suppose at some turn n, I read a file and pull it into the context. Now at turn n+m, I discover that the file has mutated on disk. The most energy efficient option might be any of these:
1. Replace the content of the file in the cache
2. Append the new file to the end of the cache
3. Add a thing to the context saying the file has been updated.
4. Compact memory then do option #3
As a harness builder, it's pretty hard for me to pick the most energy efficient strategy - I assume every model has a different cost for prefill vs decode and I also expect that a provider (an inference cloud) might have slightly different implementations that would impact how expensive a prefix-cache miss is.
Ofcourse, I also need to target aggregators like openRouter because as a solo hobbyist harness builder, it's really hard for me to track every cloud's unique API. Then there's the fact that I'm optimizing for lowest billed cost, not lowest energy consumption.
I'm waiting for the industry to standardize on a good way to express these in an API so that I can start doing these tradeoffs at runtime in my harness.
It's strange that people who have LLMs write entire articles for them often claim this. It might be true that you prompted it with your ideas, but the result is still in its own style, and style (and it variety) is why we prefer to read human text.
If this is in relation to removing the need for shell, my idea is more like inspecting common files like Makefile or package.json to extract and dynamically generate tools for the llm call.
I recall this company now for their unintuitive/abstracted pricing model I have no interest in, but it does make clear why they would be helping customers with cache hit rates, that's going to lower bills on a per-request basis