GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
Designing for extreme expandability comes with pretty steep tradeoffs.
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
Maybe a dumb question.
https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate
I.e. low memory bandwidth.
They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
4.4k seems... I just realized I have no sense of scale whatsoever
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
or 40 macbooks with each 4 ssd. to get 40 tps.
it is even worse with 40 macbooks.
if 40 macbooks is all that take to serve a 1TB model with decent speed then you would see everyone selling the models for very cheap right now.
Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with
Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier.
What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times.
Numbers in results/SCALING.md and results/PREFILL.md.
Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.
Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent.
Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall.
Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.
Kimi K3, 2.78T parameters, ~1.45 TB of MXFP4 experts streamed from four SSDs on an M5 Max / 128 GB. 1.00 tok/s steady over 512 tokens, 1.13 over 128, ~6.3 min to first token on a 512-token prompt. Output token-identical drafter on/off on a given drive layout; the int8 trunk is non-weight-exact per upstream.
The useful bits: one drive gives ≈52% of four, two ≈73%, three ≈90%; and prefill is slow because of ~9 TB of reads for a 1.4 TB model — a scheduling bug with a planned fix.
README with per-run logs: github.com/argonautlabsai/deltafin — a fork of gavamedia/deltafin, who built the engine.Quick question - does it really need external SSDs, or if the local SSD fits the whole model - how fast the model would be? e.g. on your machine, M5 Max 128GB, with 4TB SSD? maybe it'd be good to add "0 external SSD" column on your graphs?
so... enough for 1 question + 1 long answer?
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
"We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard".
These demos are maybe useless, but if open models keep progressing, there's going to be some break through that continues whittling down just how much needs to be kept in VRAM, and progressive degredation to regular system ram and to ssds.
Afterall, they're not writing anything to these, so saturing all bandwidth could bring models to the masses. all without any help from Zark Muckerberg.
Doing something like that at 1 token per second may be useful.
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
It is cool that they got it to work at all.
Such of a Claudism. Not criticizing, just noticing.
The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs
(we found the hard way that squeezing those makes the verifier reject wide batches and decode falls to single-token steps).
With the current reserves the engine admits ~4.4k tokens; that's a configuration ceiling you can raise by giving the cache more of the 128 GB and accepting less headroom elsewhere.
K3 itself supports far longer contexts — but see the prefill caveat above: on this setup long prompts cost minutes per 512 tokens until the scheduling fix lands.
So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals.
Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices.
In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower.
In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes.
A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput.
Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
> Just imagine someone asking this question today.
to be fair, considering how young folks live whole life in a smartphone nowadays...
(0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load
https://raw.githubusercontent.com/argonautlabsai/deltafin/ma...
If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.
Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB);
the drive behind the hub reads 5.7 GB/s and falls with queue depth.
Details in the README's hardware section
I wanted to see whether Kimi runs at all on one machine with the full record published, and for long multi-table finance reasoning I wanted the strongest model I could keep on the machine.
I did some tests against Deep Seek v4 flash results on my reports and Kimi definitely has some advantages.
RAID0 works fine only when all the interfaces have the same speed.
If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD.
In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs.
One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1
This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
So I'm extrapolating this same idea to LLM inference.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
Which tasks? I think you're underestimating how token hungry current proposed workflows are.
You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s
I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something.
Its not a problem of the model of course, but it seems impractical atm.
I wonder if anyone else had this kind of thing happening.
I ran opencode task again last night since it's slow and this morning i got a "SSE read timed out"
logs show tokens still streaming in steadily (every 3-4s, 46,818 tokens in) right up to the moment opencode disconnected
opencode cut a response that was actively generating (not a stalled one)
But it's only possible to make custom ASICs when you have a specific problem to solve. For newer crypto systems they can vary enough parameters that building a flexible enough ASIC to recoup the investment before the algorithm changes and makes your hardware useless.
For problems where the problem to solve remain in the problem space the ASIC can solve there is no point to use a thing else.
hermes > pi > opencode ??
or is hermes better in general? habent tried it hermes/pi yet
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
- list of many endpoints with the JSON they receive and return and description of what they need to achieve
Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy paste errors or dumb stuff I'd need to fix before it's shippable.