A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.

However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?

A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.

[1] https://github.com/1stproof/batch-2/tree/main/batch-2-submis...

Any benchmarks showing if this actually improves problem solving? Or reduces errors?

The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.

So I'm a bit skeptical!

And the associated GitHub repo: https://github.com/DataForScience/LLMs

> The plan is a graph

I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.

That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.

Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).

Good stuff, keeping it

why do i hate skillks, harnesses , memory systems whatever. such ideas that everyone thinks they've discovered but are totally useless in practice.

I'll take the "best way to elicit a clarification response on the internet is to state the opposite confidently" bait...

The example listed in the article -- fanning out a few simple get-population, get-timezone, and make-summary calls -- is, in fact, useless overengineering. This is a basic promise chain with extra steps (priced with tokens).

But as with all software pattern learning, we learn the concepts with simple toy examples that generalize into something bigger. It's the generalization that matters here.

This is talking about a few methods and tricks for spawning effective subagents (collectively, that's the "harness"). Those tips and tricks are nice, but to not be considered useless, we need to make sure we understand why spawning subagents is useful in the first place. Yes parallelism is nice for some tasks, but that's not really what this is about.

The real reason is protecting your context. Yeah, we have 1M context windows that can fit all of LotR in it, but these machines work better when they're narrowly focused. Large context windows run into attention issues and forgetfulness ("Yes, you're right, it was stated I should/n't do X but I ignored it, my bad."). So subagents come into play when you don't want all the tokens associated with a subtask to pollute your main/primary context window and degrade task attention. Split that off to a subagent, let that context navigate the details, and just make sure your main one gets just the input/output blackbox results.

The trick is getting a sense for when the complexity of the task warrants that kind of context protection, vs when a single agent is good-enough. Your toy example will never have enough complexity to warrant the setup, but you might one day find a generalization that may.

No clue?

I don't think they are totally usesless.

And its clear that progression is happening on a communith level on all of these and they get integrated later on in commercial offerings like from Anthropic and co.

But also doing a opensource harness and not just giing in to the big companies allows us to have all of this open and transparent and with open models locally.

What each of these is doing, fundamentally, is solving context management in an opinionated way (that and guardrails).

agreed. All of these are trying to get to something that can't really every be achieved with LLMs with is determinism. Folks are trying to constrain the models to behave in a certain way with all of these tools, but there's far too many edge cases for them to be reliable. Doesn't mean they can't add some value, but it seems very limited. Hoping it's only a matter of time before we go back to engineering and step back from "vibes".

Because its all moonshining.

Trying to make gold from pyrite.

so if you don't use skill, harness and memory systems, what do you use?

Wrong. They are commonly used by millions.