The Harness Is the Thing(scott-fryxell.github.io) |
The Harness Is the Thing(scott-fryxell.github.io) |
Good thing then that you're fully in control of your own actions.
A gift granted to us by being a fully grown adult that is also likely registered to vote.
Meta: There is of course a way to put this less snarky, but that doesn't slap people in the face as hard as they need to be slapped in the face to maybe one day start remembering that they have agency.
I cancelled my Anthropic subscription until they fix how their models write and it's no longer unbearably annoying and obnoxious. The concise output format is a step in the right direction but I need a few months away from them.
Kimi and GLM models on Max reasoning feel pretty close to Fable. That said, even at their most expensive plans, a single one of them might not always be enough, while getting both of them for a year gives you a nice discount and isn't insanely more expensive than Anthropic. The problem there is that they could still easily rugpull you with token limit changes later, I don't trust any of the big labs not to mess around with those for any length of time.
Also most harnesses let you choose models per sub-agent. Like I can use Fable for running the main session and just tell it to use Opus agents for implementation in Claude Code, same with the Kimi and GLM models inside of OpenCode and other harnesses. The only problem is that the UI for controlling sub-agents usually really sucks.
Also, OMP has a solid subagent model and I like it with some tweaking.
For me, and my projects, it’s been great. It’s made an enormous difference.
I guess my workflow may seem “quaint,” to many folks, here, but the end results speak for themselves.
I suspect that one vocation that could get heavily impacted by AI, is the consulting business. That’s where many experienced people go, as they reach their career peak.
In my last project (just about to ship), ChatGPT replaced a whole bunch of services that would usually be supplied by external advisors.
But these are also services that I would normally not be able to afford, otherwise, and would just have to “make do” with. This release will have a level of polish that I have would never been able to achieve, unassisted by AI (I had originally used “on my own,” there, but the reality is, it actually was “on my own”).
I don't have Fable at work but I'd probably use it for actual code if I did because not having to spend time handholding the model on this stuff and getting useful code first try is very useful
I feel the value I get far exceeds $600/m. It's a straight expected value calculation for me and I'm happy to pay. I wish they'd make it easier though - just sell me a 100x account for $1k/m and save the messing around.
And Opus5 aggressive audits.
Once it has exactly your coding conventions and access to other code to copy bespoke patterns, a strong idea for what to do, then you can let it do the work.
You do the wiring, it fills it in.
Coding was never the work.
Beyond that, I find this whole plan and build thing to be a pointless waste of tokens. If your planner made a detailed enough plan, then the cost of executing that plan is a just one turn more of cached tokens, and minimal time.
Meanwhile: switching agents, reloading context and building from the plan will easily balloon your token use and time. And any emergent problem that the dumb executor finds will instantly wreck the implementation because they're not competent at solving it. And if your plan is so perfect that there's no edge case then you're wasting tokens because your planner was one turn away from finishing the project via cached tokens.
Yet the simple blog website static page saying that looks very weird and broken on the desktop firefox.
How large should be a development team to make proper margins in 2026?
Normally I'd just switch into FF reader view when it's that badly done but that doesn't work either.
When I, as a single person, can produce a project in one month that would have taken a team of four people three months to produce, why would I care about token cost? I’m now spending $500/month instead of $40,000 month to get the same thing 3x faster. $500 for a project instead of $120,000. (Assumes my cost, $40k is the other three people)
It’s a no-brainer —- use frontier all the time.
He never had to hire anyone in the first place.
He didn't have to make any contracts, deal with screening, background checks, recruitment calls.
All that money he didn't spend he can spend on maxing out token usage.
And our 2 non-technical staff are now busy designing apps, so when they hand off something to be productised, it’s far more complete than the old days of a few Figma drawings
-- CPG Grey, Humans Need not Apply. Released over a decade ago.
Agreed. You may think that the task is simple, but a brainfart of a dumb model overlooking something can cost you more in time and effort (if you relied on the output of the wrong code and now have to go back to regenerate), that it just doesn't make sense to use non-frontier for anything but hobby projects.
(I also tried the "let a massive model make massive changes" approach and am still psychologically recovering from the experience. The codebase may never recover!)
Also, Luna and DSV4 Flash seem to be on par now except Luna is faster and cheaper?
The exposure to Deepseek made me question the valuation house of cards built on SOTA providers. There are more companies producing competitive and useful models than there are companies producing jet engines for airliners, and not for the lack of trying. China has been trying to make these engines for decades and so far failed (their flagship C919 airliner is using CFM, American/French), but it has produced at least three competitive model companies within 3 years even though they're handicapped by their hardware.
It's simply not that hard, and diminishing returns will, in fact, diminish.
Why jet engines aren't made in China
https://news.ycombinator.com/item?id=48740971
That's a very interesting point of comparison though. I wonder why that should be the case? Why is it so much harder to make a jet engine than a language model?
Maybe with LLMs the iteration times are lower? Or there's more public information about technique? Or are jet engines just intrinsically a harder problem?
In each iteration you make an LLM call, perform some work (e.g. tool call), augment the prompt (append or compact etc.)- not necessarily in that other- and continue.
Until an end condition is satisfied. Then you break out.
This example has one, but if you add a second one around their example code, and take user input, and there's a basic harness already.
When you have a few example chats you want a model to emulate - say you made it from your proprietary data, you can train any open model on that data in this cheap way. You don't lose any quality versus not using lora since the models overall knowledge won't shift that much due to your data anyways, so it's a waste to make high dimensional updates.
However, only in some cases is it worth it and equal in quality to just making a good retrieval system and exposing it to claude code or whatever. If a retrieval system over the same data is very difficult, or if the data simply must be proprietary, then you should go for it.
[1] technically "rank", but I'm simplifying
I use Claude plan mode to do relatively small changes and even then I find that if I actually try and think through the problem and solve it myself that I find good metaphors that will aid future work, and I will discover tangential issues which are then important to look at.
I wonder if we need a list of things to tell AI to stay sharp, like this one. Sometimes I tell the model existing design is stupid and then it explains reasoning to me.
It will save you a ton of time and you won't be tied to fable costs
[0] Search, Plan, Assert, Code, Evaluate. Obras superpowers, Matt pococks skills, and the nori high autonomy skillset all have this built in
It would be so much safer.
The harness says "You have access to tool X, Y, and Z, but not A, B, C".
The sandbox says "If you try to use X to access a forbidden resource, I'll prevent you from reaching it".
They don't sell cars without seat belts, and it is the car manufacturer who has to do it.
Also, small nitpick but technically a container doesn't give full isolation compared to something like a VM.
"permission": { "bash": "deny" }
Or something equivalent in any agentic editor of your choice.
the pattern of layering of deterministic, probabilistic, deterministic, probabilistic. it's a strange pattern but it seems somehow natural.
I generally use larger models to plan. All my generated Epics have similar structure. All my repos have similar structure (https://github.com/brainless/akar and https://github.com/brainless/daftprompt are recent examples).
I barely spend time or thought in making prompts. I have a simple text file with a few combinations. They refer all the common files (README, AGENTS, DEVELOP, etc.)
All reference software is cloned locally and the docs mention that. The prompt templates then boil down to research mode (write Epic) or worker mode (write software) or review mode (leave review notes in Epic). That's it.
Many of my harness experiments are about text manipulation, text search, graph on text. Because that is what LLMs are - text processing systems. Cut parts of prompts, cut parts of response, cut parts of user's intent. Join, break into epics/tasks, run with LLMs, repeat.
Plenty of devs developed anything from operating systems, enterprise software or videogames on their own.
I still see no evidence that this has changed significantly.
... but clearly AI was used in the development?
That seems deeply cynical.
> Single developer projects can build to the caliber and consistency of large development teams.
This has always been true. Good developers, like truly good devs, could run rings around a team of mediocre devs. It’s a multiplier, a team of 10 1x devs will get dominated by a single 10x dev no matter how much AI they use. Nothing has changed here, if anything it benefits the good developers.
> At the moment my rig is supported by two subscriptions (Cursor, Claude) that I can augment with Pi as needed.
This conflicts subscriptions with an actual harness, doesn’t bode well for the rest of the article…
> Recently I learned about prewalk, Can Bölük's technique that uses frontier for the planning phase and first task, then hands off once the pattern is set.
This pattern has been known for years and is not attributable to a single person.
> Exploration leads to a plan formalized into an explicit DAG (directed acyclic graph) task list. Then a worker takes over, focusing on implementing the DAG one node at a time. Once complete, I bring in the critic to simplify and question what was implemented. Often this phase will push back enough that the worker phase is revisited. But once satisfied, the critic gives way to a promoter, which is my reminder that a job is not complete until you've properly communicated it to others.
I’ve tried all these complicated workflows. In the end the best way to use LLMs is to give it some instructions, take a look at the code, and then ask it for changes. At the end, ask it (in a fresh session) to review the changes for bugs or incorrect assumptions and architecture. Rinse and repeat. Anything more complex is over engineering.
None of the rest of the article seems particularly interesting. Just more busywork.
> HUMANIST SOFTWARE DEVELOPER
Uh huh?
I have LLMs align stuff for me all the time because I'm too lazy. It takes the screenshot, changes the code, code auto reloads and boom, done.
If everyone is doing the job of hundreds, extremely productive and everything is so easily fixed, why everything feels so slow and broken, even so basic things? That's kind of the point. I'd expect nearly perfect websites everywhere, especially from "productivity" people who mastered the flow.
Excellent feedback, Thank you.
It's not
For general workflows, I agree. Doing some DAG system is just a waste of time if you don't have a grounded verifier for every node. If you are going to human verify, why make a dag of subagents and waste time? Keep the dag in your head, the way we all did before LLMs. The dag in your head is also much better.
For specific workflows however, even this is under engineering. If your specific task or family of tasks can truly be decomposed into multiple verifiable subtasks then you should spend the effort to build that system. It doesn't matter if it's technically worse than the other method because it is basically automatic and incredibly cheap. For complex systems this is a lot of upfront work, but if it is done, then the return is completely outsized.
I painted it black and white, but you can of course compose everything.
I tend to work in 5-10 or so parallel streams at a time so that also multiplies token use as a function of time. Much less and I'm waiting for it too much, much more and I can't juggle effectively. My new big project is of course to try to take myself out of the equation further and increase the parallel streams dramatically; pretty hard to get that right so still manual for now.
But I think this will eventually be a problem solved at the OS level in a more streamlined way. I.e. there will be fine-grained permissions you need to approve to give an agent access to the system.
This is my first blog post with graphics. so your comment helps me guage proportions
This doesn't make sense. There were slow and broken things before AI. This was true even when the things were made by hundreds or thousands of engineers.
If an organization doesn't care about making their app or site fast or correct, then unless they have a truly ludicrous amount of free manpower (more than is available now with AI, because review and architecture are still bottlenecks), it probably won't happen.
AI doesn't change this. AI doesn't change the priorities of an organization, it just changes how quickly and cheaply they can build. And every org has a point in their priority queue where things are no longer worth it to spend resources on.
If Venmo didn't care about making their login page work well with Firefox (which, in my experience, they don't) before AI, why would you expect it to be better with AI?
(I'm in the second boat so long as I'm responsible for the code I PR)
Is there any alternative model with design sensibilities?
If you're happy to "lead it by the nose" you can do well with a lot of very low end models.
If you want to kick off a "/goal run until [complex verification passes]" and let it run for a week with minimal intervention, then not so much.
You can make do with cheaper models for long running agentic runs too, but it tends to require a lot of extra scaffolding and additional review steps.
The overall layout is fairly odd & weird - but it's the same in Chrome & Firefox. There is one technical bug with the main body font-size - it uses some invalid syntax (should be invalid in both Chrome & Firefox) & Chrome seems to be accepting it (against spec). The rule:
font-size: clamp( 1.125rem, 1.125rem + (1.333rem - 1.125rem) * (100dvw - 24rem) / (80rem - 24rem), 1.333rem );
Try laying some people off just to be sure.
"then the cost of executing that plan is a just one turn more of cached tokens, and minimal time."
This is just not true at all. There's a huge gap between 'figured out the hard stuff' and 'rock solid'.
Dependencies, integration, corner cases, docs, testing, unforeseen issues, a lot of back and forth auditing making sure things are really tight.
Audits get diminishing marginal returns, but you have to do them until they don't find anything, and that's usually a few cycles.
So aside from the fact there is 'a lot of labour' - part of the plan (maybe the most important part) is documenting most of the trip-up scenarios. If you ran an experiment or two in the background your agent will 'discover' a few key odd things, you back those into the plan.
I'm 100% certain that this pattern works because I (and others) use it very successfully.
Hint: save your main context by using sub-agents to do grunt work - even in impl phase - farm out anything directly implementable without a ton of background.
Also - make a skill so your Claude can call Codex and visa versa and maintain long-running sub agents of 'the other kind'.
An Opus with 1M context window executing on a 'plan' that a Codex 'sub-agent' is executing on - ad a different Opus sug-agent is auditing hard ... that 1M token window is dramatically extended to 'many millions of tokens'.
That can work within Anthropic/Codex Pro plans.
I don't work where we can ship slop. I don't work where PRs can be merged based on what the agents say. I work where a human has to read and approve and own every single line of code. I work where the stakes are actually high, so the cost of not using the best tools in terms of human time are big. A single turn around in a PR costs more in human time than the difference between deepseek and fable in API costs.
So, when you admit "There's a huge gap between 'figured out the hard stuff' and 'rock solid'." but then claim that the cheapest/dumbest agent in your arsenal is your go-to for "rock solid", I have to question the quality of your results.
Personally, "using plan mode" is a very 2025 way of using these tools, and I wouldn't be surprised to see "plan mode" be removed from codex/claude code/et al.
Realistically, I'm using the best models to think about a domain and problem (Fable High+), and I'm using a cheap daily driver with an advisor pattern (Opus High + Fable) to iterate through POCs, and I'm using human review to guide design. None of that is "plan mode", it's actual engineering. Then we decompose the solution, we stack it, and we use only really strong agents to build, review and refine.
This obsession with cheap agents leads to low quality outcomes. "Rock solid" deserves the best tools, and the "plan" will never be good enough. I'm going to be sending fable xhigh and sol 56 xhigh et al at it in adversarial review, why the heck am I cheaping out on the actual implementation?
And finally: my time costs way more than any of this. Cheaper models are slower overall and when combined with re-work time, are dramatically slower. I'm costing my company hundreds in my time to save a few bucks on the API bills. Nonsense!
Based your arbitrary dismissal and unwillingness to even try to consider new patterns with which you may be unfamiliar - it may be difficult to communicate with you.
I have the advantage of 'certainty' because I have the evidence over many projects / team members.
We ship near perfect code.
In addition to the hints above, we do this at least in part by explicitly anchoring and testing requirements into several aspects of the code, and ensuring that known 'weak spots' are managed.
The 'planning process' ensures the requirements are mechanically anchored and integrated into tests, that 'proportional' documentation is applied, and that module, library and project level documentation is perfect (and mechanically validated where possible), which FYI is what solves most of 'context problems'. (That's another hint, if you have extremely good docs, you don't need to load vast amounts of code).
Yes - I hear you that 'time matters' and that 'the stakes are high' - consider that you may be talking to people where the stakes are just as high, or higher - but more specifically, this is not about 'saving tokens' or cost so much as it is using the right level of model for the task.
Use the best models for background research and planning, use mediocre models for execution, and mid-high for auditing - in other words 'use the right model for the right work' - and in a certain methodology, dumber models are appropriate.
FYI this saves you the ugly 'Fable' problem which many are encountering as it burns though Max plans. Don't 'automate' with Fable, it's the wrong model for that.
I could go on, but consider that there are actually ways of organizing projects and orchestration that work well.
But in my case, it wasn't nearly so exotic. The LLM helped me to do a much better job, preparing the App Store presentation, Web support, privacy policies, budget prognostication, and app glossary.
I have just had an extremely complex app, pass App Review, in record time (from going into review, to approval). No niggles or bounces at all.
I'm thrilled.
Basically, my needs are different from others. I'm not working on the next NORAD upgrade, much of my work is open, and the more ChatGPT knows about me, and the app I'm designing, the better. One reason I chose it, was because of this "memory."
TL;DR: I feed it just about every scrap of information about my project as I can. Source files, documentation, screenshots, videos, information about the organization, information about the target demographic, etc.
With all that information, it gives me very useful advice.
It created a great tutorial. I usually write way too complicated ones. It did much better.
In the case of the App Store stuff, it helped me to choose the right privacy report, generated the privacy manifest, and helped me to compose all the copy on the storefront.
I'll probably be releasing the new app, soon. It's already passed review, but I want to make sure that everything is kosher, before releasing. It came together so quickly, that I have the luxury of time. I just need to release before (or as) iOS27 comes out.