Qwen3.8 27B scores 52 on Artificial Analysis(artificialanalysis.ai) |
Qwen3.8 27B scores 52 on Artificial Analysis(artificialanalysis.ai) |
If tps is so high, a compaction step could be performed over every thinking turn to keep context size down.
How so?
stlouisfed said AI companies accounted for around 4% growth of US' GDP in 2025. Not 2008 catastrophical, specially because there's little public debt involved.
The actual productivity boost from using AI is there, but its diluted in slop + it's not like AI will disappear if things go south. Chinese models and discounted GPUs will be great for local setups and startups.
Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B).
Sources:
- https://artificialanalysis.ai/models/open-source/small
https://simonwillison.net/2026/Aug/16/qwen-38-27b/
It seems like the token usage is 2.3x GPT Luna Max and almost 2x Kimi K3!
I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
On the other hand, there are some of us who are stuck with hardware that has plenty of compute, but limited (V)RAM. The new 27B is just perfect for that.
xhigh -> "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer."
medium -> no mention of effort (sentence omitted)
low -> "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
in my testing this doesn't seem to produce exactly deterministic thinking levels, because it's just a system prompt nudge. i had instances where medium thought longer than xhigh
It's a dense model so it will use all of its parameters per token. 37B active parameters isn't tiny at all, it's almost what Deepseek R1 had, and it's 2/3 of what Kimi k3 uses, so it's not going to be “insanely high” tps: it's going to be three times slower than Deepseek Flash (Prefil speed is going to be quite high though, but not token generation).
The fact that we have a GLM 5.2-class model that can run on two 3090's comfortably at Q8 is absolutely insane. It wasn't long ago that GLM 5.2 was considered amazing for open weight models.
https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B
And then a Bonsai ternary on top of that model.
But I think the next step will be even more thinking on smaller models. Maybe fine-tuned and we get really crazy stuff
It's both funny and a bit terrifying and I still can't quite believe it. It runs decently on a gaming PC! Opus 4.6 came out only 6 months ago and was then broadly considered the new SOTA by a comfortable margin! How in hell did they package capability in the ballpark of a Feb 2026 frontier SOTA into 27B?!
More importantly, what's the point of building monster-scale data centers on unprecedented amounts of debt when a more than good enough model runs on a GPU from a couple years ago?
The coming months are going to be exciting, that's for sure...
It gets really agentic at the higher reasoning levels. It does the basics like goal tracking and tool calling well, but more than that, it gets obsessed with solving problems and will do insane/unusual things to get to the solution. It actually reminds me of GPT-5.6-Sol-max which is similarly obsessive.
It doesn't surprise me at all that it outscores Opus 4.6. Opus had way better world knowledge but was more "human" with agent stuff - sort of lazy and uncreative, basically giving up once the obvious solutions failed. These newer models work more like magic, they are so creative and persistent at finding ways to get to the solution.
And I just can't fathom that the new 3.8 beats the new DeepSeek V4 Flash (which, in my eyes, is one of the best everyday coding models).
What an insane release, and convenient size to use every day/locally.
but i will test this model extensivly.
Were your opinions effected by the harness ?
DS is an amazing combo. It probably could only happen in China, not in current USA or EU (for different reasons)
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
--- (benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
Open Router
Input /M $0.45
Output /M $3.20
Cache read /M $0.05
Throughput 27 tps
It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?
I would really like to see some open source US companies out there.
We will see, since if true then it is likely the other makers of small, dense models will copy it and include high reasoning by default.
If that also makes the other dense open source models better, then you are probably correct.
2nd, and maybe more importantly: KV is not as efficient (vram usage-wise) as something like deepseek v4 flash. for 256k, fp8 kv is 9.3gb (full precision ~17.3gb). deepseek v4 flash is ~2.5b for the same size at full precision (which is fp4/8, if you are interested in it, read the paper, its pretty cool).
Doing the math, hosting 27B at NVFP4 (~23gb) with 2.3M total ctx (9 agents) matches the vram usage of ds v4 flash for the same 2.3M ctx (2.3 agents). the break point is 1.5M (6 27B agents) if you use full precision 27B.
To be clear, the qwen3.5 architecture (what 3.8 uses) is still considered decent in terms of KV efficiency, its just that dsv4f's architecture is SOTA in that space, and with the lower active params, you get better max kv scaling and higher speed serving that, if you have a lot of gpus.
(There's so much voodoo to understand—which ones are MoE, which are better at agentic tasks, etc. Versioning isn't as straightforward as it used to be.)
This is what is happening to the memory supply. The only barrier to running AI, powerful enough for most tasks, at home is the memory supply has been bought up by a handful of AI companies.
To my surprise, providers on OpenRouter (io/akash/chutes) are serving Qwen3.8 27B at ~ $0.4 (in) / $3 (out) / $0.25 (cache), more expensive than DeepSeek v4 Flash.
https://openrouter.ai/qwen/qwen3.8-27b / https://archive.vn/RrDGO
US companies have spent hundreds of billions on their models, and they are not much better than the cheaper open Chinese models.
Perhaps they can out-compete them. If not there will be increasing calls to limit access to open models on the grounds of "safety".
Basically, if you can't beat 'em, ban 'em.
Unbelievably cheap and remarkable technology is coming. History repeats.
What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
Don't forget `mistral-small` (Apache's LLM), which to me is equally as impressive as qwen3.5 (only benefit of qwen is seeing the pre-text reasoning is often more helpful than the actual text output).
...or did Prism do something special with their "bonsai" releases? I didn't notice anything like QAT being mentioned.
You can also apply fixed token budgets for the reasoning blocks, though it will decrease quality in some cases.
This happens in The Three Body Problem as well. A second ship is sent after the first and surpasses it. It's a great series.
Probably because the future "monster" models will be insane. 100T+ param models might be the type of things that can independently run a small business, which means anyone not using them is at a distinct disadvantage to their competitors.
The top model from 2025 looks silly compared to the top model of the first half of 2026. Do you feel like progress has stalled?
I do. Pre-training is where the industry saw the “emergent properties” of LLMs arise and, for a time, people thought you could just keep scaling up bigger and bigger models but then the incremental gains from doing this did plateau. Labs will still do bigger models (Bytedance has a 10T planned), but these are sparse architectures and they aren’t going to have mind-blowingly greater intelligence. Fable didn’t either.
Scaling pre-training tokens also flattened out. What is still delivering gains is scaling RL on verifiable tasks. But that’s not general intelligence - it’s fitting models to specific tasks, which ML has always been good at. More importantly, most tasks to which humans apply their intelligence don't have computationally verifiable answers.
https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
Very similar performance to 4.6 and codex 5.3 but slow and token inefficient. Still wildly impressive. Initially I didn't believe the results because 3.6 27b couldn't complete the benchmark so this is a massive leap in capability.
I'm running an RTX 6000 Blackwell. It regularly spent over an hour per turn thinking. Every time I looked at it, the thinking trace seemed coherent, sensible, appropriate. But it could never settle on a solution.
Granted, I was trying to have it solve a hard problem that 5.6 Sol couldn't solve, but still.
Either way, I'm still impressed. It genuinely feels better than Sonnet 5
> Granted, I was trying to have it solve a hard problem that 5.6 Sol couldn't solve, but still.
> Should I verify their README changes? They didn't ask me to. "I've added some notes in the README. Thanks" — that's a closing statement, not a request. Reading the README unprompted to check their notes could be seen as helpful diligence, but they didn't ask for review. Keep it simple: acknowledge, brief close.
Per Artificial Analysis benchmarks, Meta's Muse Glimmer 30b (open weight) holds its own (for agentic code workloads) against models 5x to 10x its size, too.
Once you send your benchmark to "cloud", I don't think you can rely on it being secret/private any longer.
Which harness for the benchmark ?
You have previously commented on using OC/GLM. R u going to stock with it?
pi, with a plugin to do web search / web fetch.
> You have previously commented on using OC/GLM. R u going to stock with it?
For personal use - probably yes, z.ai + kimi + opencode go subscriptions, with some share of local models now. For work - claude code, codex.
3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas.
It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size.
With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware.
Same score as the latest DeepSeek Flash 0731 which has 284B parameters! (13B active)
Its also the second best Qwen model, much better than Qwen 3.7 Max, but significantly below Qwen 3.8 Max.
I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe?
So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one.
I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage.
I will definitely give 3.8 27B a better shot than 3.6 though.
Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.
More like 3-6.
Qwen 27B full quality is FP16. So 54GB. In practice most inference providers would serve FP8, so 27GB.
DeepSeek V4 Flash in full quality is mostly FP4. ~167GB official release.
So Deepseek has 140GB model size overhead... which is shared between 100s of users single inference node serves, so not even a gigabyte of VRAM per user.
Memory required for 200K of context per user:
V4 Flash: 1GB.
Qwen 27B: 13GB.[0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...
I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.
No? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.
Output TPS in vllm for instance:
- Gemma4 26B-A4B: 200-300TPS
- Qwen3.6 35B-A3B: 120-180TPS
- Gemma4 31B: 80-120TPS
- Qwen3.6 27B: 60-80TPS
This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
| model | size | test | t/s |
| ------------------- | ------- | ------ | ---- |
| gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 |
| gemma4 31B Q4_0 | 16.1 GB | tg512 | 40 |
| qwen35 27B Q4_K | 15.9 GB | pp2048 | 1248 |
| qwen35 27B Q4_K | 15.9 GB | tg512 | 39 |
| gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 |
| gemma4 26B.A4B Q4_0 | 13.3 GB | tg512 | 160 |
| qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 |
| qwen35 35B.A3B Q3_K | 15.7 GB | tg512 | 144 |
You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
./llama.cpp/llama-server \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \
--webui-mcp-proxy \
--no-mmproj \
--parallel 1 \
--kv-unified \
--flash-attn on \
--fit off \
--split-mode tensor \
-ngl 999 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-ub 256 \
--no-context-shift \
--host 0.0.0.0 \
--tools all \
--jinja \
--ctx-size 262144 \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--reasoning on \
--chat-template-kwargs '{"reasoning_effort":"medium"}' \
--reasoning-preserve \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
Also, I think the fact that small open-weights models are catching up to the frontier rather than the frontier rapidly pulling away is evidence of this. In fact, by far the most dramatic capability increase story over the past two years has been the gains made in the small-parameter regime.
One might think, "hey, this agentic coding thing was a pretty big deal!", but I think it's a bit of a distraction because models only recently became optimized for this specific use case. It's not like they suddenly gained so much general intelligence that they magically had the ability to use a coding harness. No, the labs started spinning up a bunch of RL environments and generating rewards over long-horizon trajectories of combining these tools. It's an excellent application of LLMs but care needs to be taken interpreting how much "progress" has been mae in terms of raw generalized capability.
Training should focus on tasks that require real intelligence instead of memory. Creative writing is actually good for this if you score it on coherence instead of getting random real life details right. Basic level of coding (simple prompt to code, don't need to one shot complex projects) is also great because writing a small script is more efficient than 20 separate tool calls.
They have simply decided to not train the model in some areas such as world physics
I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.
Surely most of your use-cases are not novel tasks that combine obscure domains.
It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).
So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.
**
Analyze the 2 structural (not semantic) patterns in this text:
1. Morning revient.
2. Birds saluent Morgenlicht.
3. We suivons Waldwege toward maison.
4. Rain tombe plötzlich; we cherchons Schutz beneath sapins.
5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.
**
The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)
Can you share an example of the Greek-history puzzle prompts you're talking about?
**
The stone remembers not what the cities gave, but what the goddess kept.
Begin with the first reckoning, under Ariston.
Find those who carried the Greeks in their name and the silver in their care. Take their name as we give it to them now, in capitals. Thirteen marks.
The goddess kept one from sixty. She asks the same of every mark: give each its ordinary alphabetic number, divide by sixty, and keep what she could not take.
Each remainder walks with the next. The last returns to the first.
The first of each pair turns upon itself; the second joins it; then comes the place where the pair began.
Eleven takes its fill. Keep what remains.
Raise eleven courses beneath the thirteen marks.
In course (r), beneath mark (i), add the course to what Eleven returned. Reduce again by Eleven. Cut the stone if the result is the remainder belonging to mark (i), or to the mark walking beside it.
Count courses from zero.
The mason turns where the stone turns: the first course goes with the writing, the next against it, and so on.
A cut is `#`. Stone is `.`.
Restore the fragment.
**
Answer is:
........#....
.#.##.......#
...#.........
........#....
#.........#..
#.....#.#....
.#.#.#......#
..#..#.#..#..
.........#...
.#...........
.#....##.....
One note I had between the two is that gemma has a much higher prefix cache hit rate in general.
I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding).
Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out:
```
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve Qwen/Qwen3.6-27B-FP8 \
--dtype auto \
--kv-cache-dtype fp8 \
--enable-chunked-prefill \
--enable-prefix-caching \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \
--default-chat-template-kwargs '{
"enable_thinking": true,
"reasoning_effort":"medium"
}' \
--tensor-parallel-size 2 \
--max-model-len 250000 \
--gpu-memory-utilization 0.9 \
--max-num-batched 12000 \
--max-num-seqs 24
I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
it's good to play with harness setup where you fan out multiple concurrent branches that share non trivial amount of prefix then reduce their output/summary back into main agent.
ie. instead of serially reading further skills/relevant source files for planning/thinking, you can branch and read them in parallel reusing prefix / or use to to approach request from different angles in parallel - to map-reduce result onto main context of what's actually relevant. branching subagents has benefits of not polluting main context, shared prefix prefill is close to free on a cache hit and with concurrent decoding/continuous batching you can utilize gpu well to get good speedups.
ie. what's relevant is number of active concurrent sequences (and their shape, ie. shared prefix), not so much number of users.
i'm not sure with llama.cpp vs vllm regarding concurrency – llama server has multiple server slots, continuous/dynamic batching enabled by default, prompt caching (also on by default), ram prompt cache, context checkpoints, unified kv buffer across sequences etc. so shouldn't be bad, i guess would be good to actually benchmark. personally i'm happy with llama.cpp.