Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
They could have 9 year old AI and still post profits.
Not sure if it's my pixel or android, but I made a randos jaw drop with what the crappy AI on android can do.
When are we getting android OpenClaw?
*(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)
Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.
i have written about this:
"For device makers
Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."
https://try.works/role-model-the-case-for-a-model-routing-pr...
It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.
The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.
And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.
Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.
Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.
Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
What are some examples?
It also mostly passes the "schlong" test
Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
AI companies constantly update/change stuff, new models come out, new requirements, etc.
But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
"You are not prepared" --Illidan Stormrage
In Aug 2025 you had
- OpenAI o3
- Opus 4.1
- Gemini 2.5 Pro
- Grok 4
Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.
Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).
A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?
Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.
If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.
If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.
Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?
Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"
I guess losing some customers due to poor customer service is ok if the price of customer service is right.
6 months or even a year if something goes wrong in the fabrication process and you need to update things.
If they do more standard asic design, it could be a lot longer as the design needs to be validated on an FPGA cluster, which would necessarily need to be very big for something like a LLM. Easily up to 2 years.
There's a reason chatjimmy isn't demonstrating newer models and why they only show of an 8B model.
At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
You need to be able to add|mul where the data (the weights) are stored.
Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)
Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.
I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.
I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM
I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.
....damn. It's very impressive notwithstanding its limitations.
(/s!)
The https://chatjimmy.ai demo was impressive.
Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.
This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.
it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.
I can finally have my own Dixie flatline. Cool.
Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.
Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.
But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA
But either way, I think GP's overall sentiment of "delegating intelligence-saturated tasks to an outdated but fast subagent" makes a lot of sense.
https://huggingface.co/meta-llama/Llama-3.1-8B
As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.
It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
In case some did not know: also the movie (actually TV series) is finally happening.
# Neuromancer - Official Teaser ( https://news.ycombinator.com/item?id=49055037 )
The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.
It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…
They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not
You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.
[0] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity [https://arxiv.org/abs/2604.24827]
NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.
But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?
There’s no difference in the inference implementation, parameter count, or speed.
