Could this idea of reasoning through fundamentally different (it sounds like?) models be used to link even more distant models or systems?
If you can make a small leap, then you can construct bigger leaps through a chain. What else can latent space link to?
Instead of tool use through text, tool use through analog signals?
Instead of language prompts, analog sensors?
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
I've suspected for awhile that the frontier models work this way bc from a speed + quality perspective this feels pretty intuitive
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Edit: Same looping in BF16. So, probably have to wait on a llama.cpp update.
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
I'm convinced this isn't actually a thing for most people, but people mix up thinking with the translation of thought into words.
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense turbo nerds.
But I understand that isn't very catchy.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/
And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...
Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
Your argument that it is not thinking because it is not biological / like a human is like trying to argue a plane doesn't fly because it uses jet fuel unlike a bird.
What is your definition of thinking?
It then becomes apparent that a great number of things are able to be done without an inner voice. Volition seems to occur spontaneously and it becomes ambiguous whether narration proceeds volution of volution proceeds narration.
It's still unclear what the limits of action-sans-monologue, reading seems especially hard for the sub-vocalist, conversation, or planning. But who knows, it's a fertile ground for n=1 experiments.
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
Take the top lace under the cross and through the hole.
Make a loop in one lace.
Wrap the other lace around the loop and push some lace through the hole that is made at the bottom to make a second loop.
Pull both loops tight.
Usually it's packaged in a story about rabbits.
Non-technical people believing that these models are actually thinking can lead to all sorts of inappropriate uses.
The problem is that the language spills outside of the LLM space, so we should use terminology which is more generally understood in a way which reflects reality.
- It conveys the idea that they aren't tokens normally visible or part of the final answer.
- It conveys the idea that the longer the models "reasons", the better the output.
What other term best describes both of these ideas?
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
It is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
This is very hard for models to learn
https://www.lesswrong.com/posts/MAww2kXP4cGWz4M5p/steganogra...
Yes
> to prevent users from using their models in certain ways they consider unsafe or bad for their business.
No.
That's not the only reason why they're terrified of losing CoT, which makes your statement a mischaracterization - a lie.
They don't want to lose also because CoT is one of the key tools in alignment training.
> Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
"Personally"? That's literally the consensus amongst, like, everybody. Nobody in the field thinks CoT is an optimal way of monitoring. But it's the best way we have now.
Can't they just monitor the text outputs as well?
No, the core upside is that it lets you check what the model is thinking for safety/alignment reasons. Nothing to do with monitoring user activity. You can monitor user activity just as well without looking at CoT.