Nine coding harnesses vs. your laptop(nasutton.notion.site) |
Nine coding harnesses vs. your laptop(nasutton.notion.site) |
I've been so far using pi like this:
> pi --offline --no-extensions --no-skills --no-prompt-templates -nc -nt --thinking low --system-prompt "$(cat <custom purpose path for system/role prompt>)
Purpose was to reduce token usage to an absolute minimum (zero extra token) as often I use this per use API keys (and not my GLM key, which let's say, is a bit "different" when it comes to conversations).
Are there any other tools specifically designed for such tasks? Even though hax looks like the absolute bare minimum one can go while still being usable.
I wish there was also tools that would refuse and reject (or prevent) models from inserting "thinking" messages into responses. They keep appending those and that very soon that part starts snowballing on steroids.
it grew out of annoyance of dependencies on js runtimes, probably similar to you. mine additionally works on solaris and esp32.
could be interesting to collaborate!
If anyone's aware of a smaller agent than this, hit me up!
[1] https://gist.github.com/fourlexboehm/a60e4ef9306744483731cd1...
So the points about it being a well behaved unix tool, installing it via brew and it not being react are points I - love - to see.
Thank you for making it, I will definitely try it out.
A lot of people have multiple files for their injected system prompts (a.la openclaw or hermes), I think it would be a good idea to either add or modify this point to be able to handle multiple file injections (system_prompt_append_folder or the like). Fitting that shape would make it easy to compare to those systems and make it easier for people to transition from those systems to yours.
"it spreads up to 50% between nights, so nothing between the lean arms is a finding."
So I keep Pi for just one case - when I have to easily strip everything out for some work. Anything heavier and OpenCode or Claude are ones. I am sure I can make Pi behave as I've suggested above the "N harnesses within 1 harness" and I even tried but it simply started getting out of hand and using the harness started becoming the frustrating hobby.
As for OMP, I just don't understand why would anyone use that not Pi or other "full-fledged" harnesses.
The trick with pi is to extend it yourself as you use it. It’s pretty easy to do.
pi used 2-3x the tokens of codex. pi with subagent pkg used 8x-10x the tokens of codex.
I don't see how adding bloat to pi would make it more token efficient if the baseline is so poor to start with
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
The only other thing anyone is using is Qwen 3.8 Flash Next, only by memory-rich people.
Depending on which benchmarks you believe, these models (and the Ornith 1.5 finetune of Qwen 35B-A3B) are competitive at about Opus 4.5 to 4.7 level. That matches my experience in real tasks over the last few months.
Not bad for something you can run at home for a couple of thousand dollars.
It's fine for that (and I happen to have a 24GB VRAM GPU anyway since I game on the same PC).
It's neat but for me not world changing.
It's also just fun to be able to poke stuff and see what it can and can't do (but I could see how it could also become a time trap in cases where it gets kinda close and you want to fix that).
Wouldn't trust it for long form coding, but for shorter stuff it's really good.
I'd love to see a tiny, reproducible benchmark repo that anyone can drop on their own hardware and then run against all harnesses at once to compare the per turn prefix token count, time to the first token, experienced tokens/sec (and prefill), cache reuse % and a pass rate on a deterministic set of small tasks. I think it could also be useful to have some way to share results and hardware for others to compare.
uv run python benchmarks/matrix/run.py setup. i'd happily take your hardware's numbers!
“Chad” initially looked interesting but the minute I saw the ai-written markdown and giant commit I just left. I just can’t bring myself to read someone elses’ slop, regardless of performance.
If all a developer hand writes is a truthy and readable markdown document, I really don’t care if the rest of the project is vibe coded, but I struggle to get interested in AI generated summaries and docs.
Also, Alt-Left Arrow doesn't return me to the previous page (Hacker News) and I have to use my mouse.
https://m.youtube.com/watch?v=c_fQoDkULl0 (see around 8:00)
Mind, this is more of an exception, at least for my work, but still, what do you do with 32k tokens?
Edit: i resumed a quickie that basically added a 10 line function in one spot, and even that is 50k tokens...
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
i think headlong as well
It seems we need a place where harness creators can pay for that work. There are more people talking about their own harness in this post than talking about the post.
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.
i was steering chad in the opposite direction. one model & one set of silicon -> taken to the max. swap out your CHAD_MODEL and it still runs, you just leave the drafter and the kernels behind.
If it's missing guidance that would help, system prompt additions might help.
8x-10x the tokens is wild, is this for some tiny artificial benchmark? That's just too much extra for something not to be just broken.
Most things you can mess with the big effect is like, oh a thousand tokens ended up in the ~system prompt, or 10% extra or fewer work based on extra tool calls or churning through thinking or whatever.
Harness stuff if it's 8x worse that's like, it's fucked and broken, something went _wrong_.
Then I began to customise it to be as good as Claude but eat less taken. I got tired and I had not even scratched the surface. Gave up.
I finally realised, at least for me, Pi's best use case is - strip even the little "extra" Pi comes/starts with and then use it just like that if you have a task/work that is appropriate for that bareness.