While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
It supports long contexts on limited VRAM and is blazing fast.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks
Write-up: https://doi.org/10.5281/zenodo.21771064
Happy to answer anything about the scheduler or the correctness protocol.