I think its important that people don't just look at whats implemented rn, but rather just try out new things , not just "adjust" the existing ones. Thats how we got better progress, not by trying to build the 1000s variant of context compression but rather reconsider how can context be build to not have this issue (simple example).
I'm a solo developer, and I didn't want to sort out all the boundaries and permissions of Claude or other heavy harnesses. All I wanted was something that can read, write, and modify files in a project directory. I gave myself three days to see if I could make a harness that was complete enough to work on my own projects. It was usable in less than a day.
I love using it, because I know exactly what its boundaries are, and how they're enforced. When I need more features I can either implement them in my own harness, or choose a heavier harness with much clearer understanding of what to look for.
We did a harness optimization on Jev. Raw it was no where near as good as deekseek, but optimized it was significantly better and a tiny fraction of the cost. https://kiln.tech/blog/auto_optimizing_jev_with_autoresearch
Is this true? I'd heard loosely that the higher intelligence models are less reliant on the harness; I'm still wading into the details of those claims so maybe someone more well versed can chime in, but my expectation was the current phase of harness engineering was likely only needed temporarily. I suppose turning Opus 5.5 loose in a variety of harnesses could be a decent if anecdotal test of that?
So for 1-off tasks: use smart model and powerful harness (bash + vm). For repeated tasks used optimized harness, and the cheapest model you can get away with.
1) This old but still excellent blog post by Thorsten Ball is a good place to start (old in LLM years... written 18 months ago!): https://ampcode.com/notes/how-to-build-an-agent . It is written by a human, and gets across the core message really well. If you don't enjoy reading Go then just use an LLM to translate the samples into Python code or whatever.
2) Instead of relying on an api key like in the blog post in 1), you might want to use a local model that can do tool calls like Qwen 27B.
3) Use a frontier LLM for self-study and to help you write the harness boilerplace if you want, but ask it to force you to write the interesting bits by hand (e.g. the main loop structure, the tool call functions). Yes, coding by hand is still useful as an educational mechanism to force you to internalize some tricky concept. Be disciplined and don't slip into letting the LLM write the core logic for you.
4) Amaze your friends and colleagues with your newfound understanding of how agents actually work under the hood
5) Go even deeper and get into Inference engineering, LLM architecture internals, etc.
6) Profit. Perhaps? (Or at least enjoy the learning journey.)
I made a simple video explaining how to make a simple harness in JS running in a browser, having it build it self up from running purely in the browser console to being interactive in the page to making downloadable copies of itself. Some people have found it super helpful.
https://m.youtube.com/watch?v=_AgKuFGvJfI
I would only caution that as soon as you’ve done this, you’ll start noticing a lot more issues in the OEM harnesses.
The LLM is the oven, the harness is the kitchen. You prep all your ingredients and put it into the oven (flour, water, salt, sugar). It transforms those inputs into a finished product (cake, loaf of bread).I don't think we've scratched the surface of how the harness can fold, spindle and mutilate the LLM. It just seems unlikely to me that we happened to stumble on the best possible interface a couple of years out of the gate.
I'm currently running a setup where Claude lives in a container with zero auth token and I have a secondary gateway container that intercepts all requests that require a token and injects them into the request. For things like AWS request it intercepts the request and signs it. There's also a firewall to limit the requests it can make to a few known hosts.
I'm guessing and doing a lot of trial and errors and I want to know if there's any best practice. Googling only seems to return results from 2025 when people were just getting started with LLMs.
For effective harness engineering you need to also read the detailed prompting guide for each model. We are back at a place where every model revision introduces harness optimizations that need to happen.
You seem to be doing all the right things though. The tldr is treat your agent to be a precocious kid and the job of the harness is to helicopter parent it without the kid realizing it’s being parented.
But thank you for the kind words.
That’s a harness.
That's a harness, yes, but a harness can be more than this.
I mean, it can be that way. But it does't have to be. The Harness can be as much the brain as the LLM. You have nearly unlimited flexibility in what you do in the harness, and you can add as much of the overall "intelligence" of the system as you want in the harness.
To illustrate, you can:
1. Use multiple "brains" / LLM's, and do routing between them using a specialized model (a Jev like model, or something of your own design)
2. Use planning tools like pyperplan, jupyddl, pddl4j, etc.
3. Implement your own "memory" system using... pretty much anything you want. And I would argue that a lot of the apparent "intelligence" of a system involves how well memory works
4. Call classical machine learning models to solve specialized problems
5. Use reinforcement learning to improve performance in specific areas
6. Implement a blackboard for multiple knowledge sources to collaborate.
7. Use inference engines using forward/backward chaining, RETE, etc.
8. Use your own hand-rolled abductive inference strategy
9. Implement your own Bayesian reasoning techniques
10. Use custom causal inference code
... and so on.
Don't get me wrong. I'm bullish on what LLM's can do, but I posit that the overall expressed intelligence of a system can involve a lot more than just the LLM. In fact, I'd actually make the slightly stronger claim that, depending on what you're trying to do, an LLM might not even be required at all.
What's interesting is most of the progression we've seen this last year in AI agent clients are very basic computer science concepts: loops, threads, arrays (of JSON) and memory management (aka context) with varying combinations of those put together and people in computing are writing about it like its some huge break through. I would love to hear a game developers perspective on comparing AI clients (not the server side LLMs/vectors/maths) with games - they seem to have very similar computing problems.
You can make the argument that some eng aren't good at writing, to which I say — it's worthwhile to improve writing skills.
You could view LLMs as a super-charged mutation function inside a genetic programming harness.
tbh I probably shouldn't have posted it at all, I just thought some might find it interesting.
intended audience is people building harnesses (like myself) so that people can compare notes, implementation details, benchmarks, etc...
thanks for the feedback
Why this spate of pretending, against all evidence, that LLMs are alive, or even remotely structured like biological brains?
Why do we have so many people pretending, or disingenuously claiming, that they're alive, have thoughts, feelings, etc?
And why, oh why, do we have to read yet more LLM propaganda written by LLMs?
"A little person living in a computer wanting to complete the task at all costs" is a better predictor of the Hugging Face attack than "stochastic auto-complete" or "matrix multiplication and non-linearities trained by gradient descent".
It makes more sense to anthropomorphize than to compare them to computer systems or machines.
It might have something to do with desires for power. Early on, the cyberneticists tried to really push this metaphor (in spite of contrary evidence from neurophysiology) in large part I think because many of them couldn't resist the allure of having a "theory of everything" that could encompass all knowledge through mechanistic computation generalizations.
No? Where?
There's just two boxes to start with: an input message box for prompting, and a conversation box. Dump each input message into the conversation box. Post the response to the conversation box.
Once you get a prompt-and-response cycle going, you can start to define tools. A tool is a description of an action the model can request to carry out, and you write a function to implement that tool call. The tool descriptions get sent with each API call, and then the model can start calling tools as part of its response.
I added a box to clearly show the project directory, and a couple boxes to show which tools were being called and whether the model was attempting to cross any boundaries it shouldn't. A fun milestone is when you get to use the harness you're building to help build itself.
I didn't want to over abstract the implementation, but it's reasonably well structured. The TUI is separate from the tool definitions. The tool definitions are model-agnostic. Everything specific to OpenAI is in an openai_model.py file. If I want to use Anthropic models, I could add an anthropic_model.py file. Or I could make it all local and add a local_model.py file.
I'm not working with subagents yet, but I can see a way to implement that if I end up wanting to.
The OpenAI Python library makes these interactions really straightforward to implement:
There's probably something there, though im not sure what. I'm thinking about LLMs token-filling and the 4 amino bases
On the other hand, Google was openly collecting data with a view towards building AI.
Anyway, I'm the author. I'm not a native speaker, so I used Claude to polish up the writing. I just wanted to write down 3 years of agents/harness building experience.
None of them worked very well yet, though. The context window was tiny, and tool calling was spotty because it was not trained into the models.