How I Prompt(thorstenball.com) |
How I Prompt(thorstenball.com) |
In my experience, it varies widely by workload. Vision is great for some UI stuff but extremely terrible for my work, which involves visualizing connectivity and geometric stuff for road structure for autonomous vehicles. It can get super frustrating when I paste a screenshot and go "this polyline isn't connected when it should be" or "those polylines are self intersecting in an invalid way" and then the AI literally can't see what I'm talking about. It would have to then write debug code to dump out the coordinates of the polylines to convince itself that the bug even exists. Then it would often claim that it's fixed when it is clearly not.
I previously complained about GPT 5.6 Sol not being able to solve a simple "follow the lines" puzzle [1] without having to write custom Python code for it. GPT-6 Astra, however, is vastly improved, and could do it without writing Python. Still, for my work, it can occasionally struggle.
Many of his other prompt examples also illustrate this. The author is massively leveraging the architecture of the organization through familiarity. It could well be the case that even an LLM was used to set that up, but I doubt it happened in one prompt or without critical thought. To be fair, I guess he's sort of trying to point that out, but by saying "Look how lazy I am while it still works" or something like that in order to look cool, I think all he's conveying is "Look how easy this is", when it's potentially not and could increasingly cost him/his company wildly varying amounts of money based on how he goes about it. When tokens were virtually free, no one cared. This is increasingly not true. Many are drawing comparisons to early cloud architecture.
I find this is sort of similar to the problematic idea that in theory anyone can clone your repo and get a local dev environment running in a few seconds. So then you try it with a non-technical staff member and quickly realize how unrealistic that is as you run into abstraction after abstraction that is completely foreign to them.
That was the past though. I've seen people who were good at computers but didn't know how to code, make some good programs where the AI wrote the code. Yeah, turns out tokens cost money, but they're still pretty cheap.
With Opus 4.x/GPT <=5.5, I was orchestrating "by hand", and everything required multiple individually-instructed and context'd steps. I saw real value in the Dark Factory/attractor/workflow graph-of-agent-roles pattern as a way to automate this .
With 5/5.6, prompts could be much larger, but needed to be very deep and wide - high level of detail, lots of context added, lots of examples. They could pull code practices and structure from what's present, -ish, but require lots of hand-holding, and subagent teams were just OK. The Dark Factory yielded to the Software Factory, where the graph didn't need to be pre-constructed - the agent could drive the workflow. Looked like the future to me.
With the newer class of models, I find that they are able to infer/derive much better and orchestrate and prompt subagents themselves. They are Driving. My prompts are still large, but only because they're doing more - the individual prompt-parts are much like what the transcript showed, the workflow has collapsed into just instructions.
I'm still dipping my toes in here, trying not to blow my token allocation too quickly, and even though astra and fable have major gaps I am seeing through the fog a bit. The Factory is kinda falling away into the bitter lesson I guess.
I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.