Cool to see this running but if it's at 10%+ loss vs 16bit that's less cool.
Hard to understand that story from what's published here.
We did run an internal, unpublished FP16-anchored ladder on Qwen3-8B previously using the same pipeline:
MATH-500, MBPP+, commonsense-6
2.44 bpw: 70.8, 76.5, 74.3
2.75 bpw: 72.8, 78.8, 74.4
3.00 bpw: 74.2, 82.0, 75.1
FP16: 75.6, 84.1, 74.1
At 2 bits, the 8B model is about 9% below FP16 on MATH-500 and MBPP+, but only about 1% below on commonsense. The loss is concentrated in generative math and code, and largely returns as bits increase.
That is one reason this release is a 35B MoE at 2 bits rather than an 8B model: there is more redundancy to exploit. On the 35B, the worst regression is LiveCodeBench at -4.4 points, or -6.6% relative. The other five benchmarks are within run-to-run noise. You can find more details on benchmark results and their harness on our HF model card page.
We do not have a BF16 arm for the 35B because it is about 70 GB and does not fit our hardware (we did this with one 4090, one L40, and tested on a bunch of other RTX GPUs :), so FP8 is the reference.
The 35B bit-width ladder is measured in KL divergence against FP8:
2.09 bpw: 0.1041
3.09 bpw: 0.0921
4.13 bpw: 0.0892
The 2-to-3-bit improvement is statistically significant. The 3-to-4-bit improvement is not.