That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
Also, perf speedup?
So for this one, 27B * 1.76 / 8 = 5.94 GB
For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:
300GB/s / 5.98 GB = 50.5t/s.
Realistically it’s probably a bit slower, but that is your theoretical maximum.
If true, that would be a very welcome development.
And speed matters a lot for many use cases
This should work:
cd /tmp
# Get the Prism macOS runtime
curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
tar -xzf bonsai-runtime.tar.gz
# Get the ~5.95 GB GGUF model:
curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf
# Run the server, I used port 8331
./llama-prism-b10685-7dffb15/llama-server \
-m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
--port 8331 -ngl 99 -fa on -c 32768
uvx llm openai endpoint http://127.0.0.1:8331/v1 \
--model bonsai-2-27b --responses hi
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j
./llama.cpp/build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 32768 -b 256 -ub 64 --temp 1.0 --top-p 0.95 --top-k 20 -ctk q8_0 -ctv q8_0 -p "hello world in x86 assembler" -n 256
[ Prompt: 165.6 t/s | Generation: 40.6 t/s ]
[ Prompt: 95.0 t/s | Generation: 26.5 t/s ]
Am I understanding correctly that no draft model exists (will never exist or just currently does not exist yet)?
There is no draft file in Huggingface's repository ( https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/tr... ) and in the file "scripts/download_models.sh" of the demo repository ( https://github.com/PrismML-Eng/Bonsai-demo/blob/main/scripts... ) I see this remark:
if [ "$_family" = "bonsai2" ]; then
the projector ships in the same repo; Bonsai 2 has no dspark drafterhttps://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
32,706 tokens
38min 19s
14.22 t/sFor my "Please recite Jabberwocky" test the bf16 almost passes but the ternary and even fp8 versions fail badly.
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
I agree. I don’t know how they get such good results on these benchmarks because using them gives a very different experience. They’re kind of cool for doing short free form outputs in memory constrained systems, but I don’t think they’re useful as coding agents.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
I really hope they stop using PowerVR in pixel 11.
Which is within reach of some higher end consumer hardware, especially with layer offloading.
You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?
What’s the moat for trillion dollar AI companies? Access-anywhere convenience for models as good as everyone else’s?
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
The 3-bit quant is lossless based on benchmarks.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
If they did 60gb -> 6gb to Qwen-27B, could they possibly do the same to the MoE model. 36gb lossless 126B model seems like an impossible task.
~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.
The issue is I have yet to find a useful agentic local llm that I can run on this machine.
Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
`cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh`
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
Assume bonsai has the same ticks
Best you could realistically run in 32gb is like 28gb, or a 127B param model
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
https://gist.github.com/ctolsen/b2883e7cbf5e4357fa04366019e6...
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).
This is another pet peeve that I have with the way we phrase these things: It should be 9x "as large" and 8x "larger."
The way I'd usually phrase this, to avoid ambiguity, is "this new model is 11% the size (of the original)." Or, the other way, "the old model is 9x the size".
a∥b=((a*b)/(a+b)
A normal sum has the associative property:
a+(b+c)=(a+b)+c which justifies dropping the parentheses
a+(b+c)=a+b+c=(a+b)+c
It is just an exercise for the reader that the parallel sum ∥ operation is associative:
a∥(b∥c) = (a∥b)∥c
multiplication is typically defined as repeatedly adding:
a x b = b+b+...+b+b (a times b)
similarily one can define parallel multiplication xx :
a xx b = b∥b∥...∥b∥b (a parallel-times b)
The reader can verify that 9 parallel-times size
9 xx size = size∥size∥size∥size∥size∥size∥size∥size∥size = size / 9
So I just think the Bonsai designers meant it was 9 parallel-times smaller, which checks out...
x∥x=(xx)/(x+x)= x/2
x∥x∥x =(x/2x)/(2/x+x)=(x/2)/(3/2)=x/3
x∥x∥x∥x = x/4
Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.
Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.
But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
50 percent smaller means either half, or two-thirds the size, depending on Apple Marketing doing the math.
0.999 times smaller means size is nearly zero. 9 times smaller means you get negative memory from loading it.
I don't see what makes it hard to understand.
Idioms don't have to make literal sense or be linguistically/mathematically correct to be useful. All that matters is that other people know exactly what you mean when you say it.
And, pretty much universally, if I tell someone "the compressed file is 10x smaller than the original", they are going to know what I mean is that the byte size is 10% of the size of the original.
That makes it an idiom that is perfectly okay for everyday use.
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster. i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
The table claims it performs on par with UD-Q4_K_XL except on OCR.
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
https://huggingface.co/AngelSlim/Hy4-preview-GGUF https://arxiv.org/abs/2602.21233
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
-b, --batch-size N logical maximum batch size (default: 2048)
-ub, --ubatch-size N physical maximum batch size (default: 512)
This is the same. Taken literally, "9x smaller" is nonsense, but everyone who hears that phrase knows exactly what mathematical operation you are referring to, thus it's a totally acceptable way to express that you mean to say 11.11% as large.
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
Likewise if I repeatedly subtract a number 9 times to make it “9 times smaller” I can omit the subtracted bit.
Multiplying some scale by units per period makes sense and is both linguistically and mathematically sound.