Automation Without Understanding(arxiv.org) |
Automation Without Understanding(arxiv.org) |
Then, on top of that, they should be able to explain any of that, at any level of detail, whether talking to an expert or a layperson.
The entire premise of building machines that can provide intelligence is that they can be arbitraryly rigorous and do far more work then is reasonable or practical to demand of a human.
The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivion and carry on with our day.
But today I had some exposure to a SotA agentic team coding loop thingie which had been running almost hands-off for a few weeks on a (pretty serious) Win32+Direct3D-to-Emscripten+WebGL porting project - and I'm genuinely spooked at how well it all works; I mention this example because all the agents' processes involved a decently rigorous verification step: any time any agent confidently asserts something then it has to provide associated evidence, such as a unit test report, or build artefact, or external citation, and the system will spawn a new agent (perhaps using a different backing LLM) to verify the claim. I know a unit-test pass/fail isn't quite the same thing as, say, a medical AI agent confidently wrong about me having/not-having terminal spleen cancer, but the capability for a team-of-agents to be self-checking is definitely there.
----
Also, the past 3 years of AI/LLM/etc developments have taught me to never cling to any shortcoming or weakness they have because plenty of them do seem to have been solved or mitigated, either directly or indirectly.
I do find that both porting and translation projects have a much higher signal given the ease of mapping to tokens, when there is a proven working source to refer to - the source itself provides the validation. In a new project, you don’t have that validation.
And this is enforced by... another LLM? Seems like it would work alright until something is asserted implicitly and not categorized as an assertion.
https://mathoverflow.net/questions/43690/whats-a-mathematici...
“The product of mathematics is clarity and understanding. Not theorems, by themselves. [Their importance is not just in their specific statements], but their role in challenging our understanding, presenting challenges that led to mathematical developments that increased our understanding.
The world does not suffer from an oversupply of clarity and understanding (to put it mildly)… In short, mathematics only exists in a living community of mathematicians that spreads understanding and breaths life into ideas both old and new. The real satisfaction from mathematics is in learning from others and sharing with others. All of us have clear understanding of a few things and murky concepts of many more. There is no way to run out of ideas in need of clarification. The question of who is the first person to ever set foot on some square meter of land is really secondary. Revolutionary change does matter, but revolutions are few, and they are not self-sustaining --- they depend very heavily on the community of mathematicians.”
> The product of software engineering (or computer science) is clarity and understanding. Not programs, by themselves. Their importance is not just in their specific statements (lines of code in a specific language), but their role in challenging our understanding, presenting challenges that led to computational (?) developments that increased our understanding.
> ..In short, software only exists in a living community of developers that spreads understanding and breaths life into ideas both old and new. The real satisfaction from computers is in learning from others and sharing with others.
That seems to work. What about other areas of human activity that are currently being consumed by automation and "AI"? Like writing, the arts, or the sciences.
In other words, the mathematicians want more funding.
Ancaps are in for a major cope and seethe when they are forced to admit that the communists won the cold war.
Let's enjoy the ride. It might be last one!
I guess even if I end up being wrong about that, I'd rather enrich my life and grow as a person by continuing to learn and do hard things rather than become an annoying cheerleader dependent on unreliable tools.
“Money is the sign of a poor civilization” or something similar that a Culture GCU would say.
Also do you really think billionaires are frothing at the mouth for this because it's going to make money less important?
Sure, I can tell the time "without thinking" by glancing at my ticking pocketwatch, but other people spent absolutely enormous amounts of brainpower thinking about gears and friction and metal alloys that don't distort too much with temperature changes.
So it's the opposite: Civilization advances by using more thinking than ever before, at least in absolute terms, and I'd wager per-capita it non-decreasing.
We set upon end of human craftsmanship decades ago
Math is probably the easiest to reclaim given its right in front our faces going about daily life. The syntax of math is not that important; real world quantification the syntax is meant to represent will still exist. Our biochemistry implicitly operates on senses of enough food and water, etc.
Such measures are so embedded in the daily routines we live an intuition will always exist
No one is born knowing how to make a computer as we know them today. A cup half filled is obvious
It's there, but...
1. The project owner figured out a way to minimize token usage for agent claim verification tasks.
2. Verification agents used older and much cheaper models, including local models for the most trivial things.
3. They could afford it anyway; but I think it's an inevitability that the token-cost for a task will approach some limit for some quality threshold - concurrent with the dollar-cost-per-token shrinking over time as better hardware comes out.
> In a new project, you don’t have that validation.
I'm still trying to understand that part of the project's history, actually. Obviously the HTML5+WebGL+Emscripten+Etc entrypoint was a "new" project; one of the first things they did was build their own means of verification, I just don't know how that part worked-out in practice (besides the agents dogpiling in on TODO.md).