Qwen3-4B-Thinking-2507(huggingface.co) |
Qwen3-4B-Thinking-2507(huggingface.co) |
just install lmstudio and run the q8_0 version of it i.e. here https://huggingface.co/bartowski/Qwen_Qwen3-4B-Instruct-2507....
you can even run it on a 4gb raspberry pi Qwen_Qwen3-4B-Instruct-2507-Q4_K_L.gguf https://lmstudio.ai/
Keep in mind if you run it at the full 262144 tokens of context youll need ~65gb of ram.
Anyway if you're on mac you can search for "qwen3 4b 2507 mlx 4bit" and run the mlx version which is often faster on m chips. Crazy impressive what you get from a 2gb file in my opinion.
It's pretty good for summaries etc, can even make simple index.html sites if you're teaching students but it can't really vibecode in my opinion. However for local automation tasks like summarizing your emails, or home automation or whatever it is excellent.
It's crazy that we're at this point now.
mlx 4bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-...
mlx 5bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-...
mlx 6bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-...
mlx 8bit: https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-...
edit: corrected the 4b link
https://huggingface.co/lmstudio-community/Qwen3-4B-Thinking-...
What is the relationship between context size and RAM required? Isn't the size of RAM related only to number of parameters and quantization?
KV cache size = 2 * batch_size * context_len * num_key_value_heads * head_dim * num_layers * element_size. The "2" is for the two parts, key and value. Element size is the precision in bytes. This model uses grouped query attention, which reduces num_key_value_heads compared to a multi head attention (MHA) model.
With batch size 1 (for low-latency single-user inference), 32k context (recommended in the model card), fp16 precision:
2 * 1 * 32768 * 8 * 128 * 36 * 2 = 4.5GiB.
I think, anyway. It's hard to keep up with this stuff. :)
For a while it was china vs. world, but lately it's even more divided, with heavy camping on specific models. You can still get some signal, but you have to either ban a lot of accounts, or read new during different tzs so you can get some of that "i'm just here for the tech stack" vibe from posters.
The new qwen3 model is not out yet.
It's an interesting proxy, but idk how reliable it'd be.
I am running this beast on my dumb pc with no gpu, now we are talking!
Claude is not cheap, why is it far and away the most popular if it's not top 10 in performance?
Qwen3 235b ranks highest on these benchmarks among open models, but I have never met someone who prefers its output over Deepseek R1. It's extremely wordy and often gets caught in thought loops.
My interpretation is that the models at the top of ArtificialAnalysis are focusing the most on public benchmarks in their training. Note I am not saying XAI is necessarily nefariously doing this, could just be that they decided it's better bang for the buck to rely on public benchmarks than to try to focus on building their own evaluation systems.
But Grok is not very good compared to the anthropic, openai, or google models despite ranking so highly in benchmarks.
=====
LiveCodeBench
E4B IT: 13.2
Qwen: 55.2
===== AIME25
E4B IT: 11.6
Qwen: 81.3
For example, Google's inexplicable design decisions around libraries and APIs means it's often worth the 5% premium to just use OpenRouter to access their models. In other cases it's about which models particular agents default to.
Sonnet 4 is extremely good for tool-usage agentic setups though - something I have found other models struggle to do over a long-context.
That's political I think. I know several alt right types that swear by grok because "Elon doesn't give it any of that woke crap". They don't care that there's better, for them it's the only viable option.
of course, this is a politically charged subject now so fair assessments might be hard to come by - as evidenced by the downvotes i've already gotten on this comment