Run QWEN3.8 27B on 16gb Nvidia GPUs

Run QWEN3.8 27B on 16gb Nvidia GPUs(github.com)

28 points by Pragmata 5 days ago | 10 comments

toyg 4 days ago |

I'm not sure if I want to trust yet another loader with secret sauce, there are already a lot of those around.

What I'd really like is a simple utility that, given my system and a model, will tweak llamacpp to run decently (or tell me it can't be done).

davada 2 days ago | |

Its not so secretive, you can see the parameters they are using (arguments, and inference engine). I understand your point, but the only thing unusual/uncommon about it is the jargon used for LLM/AI systems.

Pragmata 5 days ago |

I've been using it for the past few days, and it runs really well!

I usually get 7 token/s using llama or lm studio, but this inference recipe runs at a smooth 80 tokens per second.

Genuinely very usable, and fully local!

davada 2 days ago | |

Btw, using llama.cpp you can achieve 55 - 45 token/second for processing/generation. if you use a qwen3.8-27B (IQ4_XS) verison, with decent quality in reasoning for coding/tool usage (with a 16GB nvidia).

I think now most of the struggle is getting a 32B-27B llm to work on 16GB/12GB card, because they are at least affordable/accessible for the time being compared to higher end models.

Recently the nvidia RTX 5090 32GB has reached price range of 7500 dollars (despite MSRP being around 2000 dollars when it was first launched). Crazy times.

kristianp 5 days ago | |

Which card are you using? I was getting about 40 with an UD q3 quant with MTP (prediction) enabled and llama.cpp compiled for my compute capability, but was very limited in the context size. I have an 4060 ti 16GB. Wouldn't recommend it as there's a tradeoff between larger context without MTP and about 18 tokens/s.

Pragmata 4 days ago | | |

I got a RTX5080 as primary and a RTX3050 (8gb) as secondary.

I always max out on context size so that means i need to split the model between both gpus + offload some into system ram when using LM studio. I'm guessing that is what accounts for most of the slowdown when i use that.

With this recipe i don't have to worry about it. It's pretty fast, and since its all running in the 5080 with max context it's stable (though i got to look out not to load anything big on the same gpu)

falsaberN1 4 days ago | |

With llama.cpp (CUDA) and a 5060ti (16GB) I get 60t/s with 128K token space. Odd you got 7t/s, did you verify all the model was loaded in VRAM? (--gpu-layers all)

Pragmata 4 days ago | | |

it was but i max out on context so it doesn't all fit with kv cache etc...

blagui 4 days ago | |

7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.

supermatt 4 days ago | | |

> 7 t/s not sure how you can use this

80 t/s - maybe you need to finish reading a sentence before you comment on it?