Shapelearn Qwen 3.8 27B (13.1 GB VRAM)(byteshape.com) |
Shapelearn Qwen 3.8 27B (13.1 GB VRAM)(byteshape.com) |
Disclarer: I'm unsing Vulkan on an AMD GC.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
The 5090 destroys both at 1792GB/s.
It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).
I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.
Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.
Yes, I'm not a native English speaker, and I'm definitely not an AI. ;)
I can't say the same thing because I just don't notice typos (mine or others).j I just assume my English is bad.
Isn't the definition of "being human" is "not to be perfect"? In French, it kinda is. We say "He/she humain after all" to mean that someone made a mistake.
Yes... I'm not english native. And definitively not an AI.
Also model with their draft answered incorrectly. With MTP it answered correctly.
Question was: "Does MikroTik CRS312-4C+8XG-RM have combo ports?". The answer is Yes.
You're basically saying "I tried rolling these dice one time, the green dice rolled a 6 and the blue dice rolled a 1, so green dice are better"
smaller models are attempting to distill the useful methodologies, not the license plate number of an obscure extras car on Magnum PI.
that said I wonder if there is a small 'trivia' model out there. Seems like the kinda thing Google would tackle.
- draft: 67.17
- MTP: 64.18
Why they used those examples? Seems strange.
I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
Since they’re not using a stable number of bits per token, they use their own naming convention.
Seems like a lot of them do, I only compare them within the same repo because there doesn't seem to be a very standard way of saying all the possible combinations/rearrangements.
DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.