I Rented a 96 GB GPU and Took Uncensored Qwen3.8 From 44 to 125 tok/s

I Rented a 96 GB GPU and Took Uncensored Qwen3.8 From 44 to 125 tok/s(aseemshrey.com)

7 points by LuD1161 44 days ago | 3 comments

karmakaze 43 days ago |

I did a similar thing running Q6_K model and Q8_0 DFlash2 (draft=7) quants:

    DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF
    incoai/Qwen3.8-27B-DFlash2-GGUF

using llama.cpp PR/commit https://github.com/ggml-org/llama.cpp/pull/27342 on an AMD R9700 (32GB)

LuD1161 43 days ago | |

wow i need to check the 32GB cards. how did you handle the KV cache. Oh i see it was the Q6 model. Did you have to cut down on context window?

karmakaze 43 days ago | | |

I used Q8 kv cache and using 64K context but can go a bit higher. The DFlash2 model takes a few gigs and using Q6_K (rather than a Q6_K_M/Q6_K_XL that unsloth publishes) saves some more. Also using Vulkan that has less VRAM overhead, but ROCm has faster prefill. Overall I'm quite pleased with it, but need to give it more agentic work. I also run an Qwen3.5-9B (another DavidAU finetune) for subagent editing/etc as well as an document embedding model (on different GPU). Using MTP reclaims the DFlash2 model memory and only a bit slower so a viable path for more context.