DeepSeek V4.1 Flash 1017 (Max)(artificialanalysis.ai) |
DeepSeek V4.1 Flash 1017 (Max)(artificialanalysis.ai) |
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
For best "bang per buck" I think Qwen 3.8 Flash Next (or Qwen 3.8 27B even) makes a lot more sense for most people - you can run qwen38fn at 100+ tok/s w/ Strata (hot expert offloading) 200+ tok/s on 2 x PRO 6000, it does almost as well as ds41f at most tasks (it's much less token efficient than other options, and it sucks at AI/ML tasks though).
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
Does deepseek v4.1 flash 1017 really take that much longer per task?
Just looking to get maximum usage for my open source projects where I'm not worried about my data being used for training.
But it accomplished everything I was looking for it to do, even if Claude perhaps could've done it with less guidance on my part/faster. I'm pretty impressed with it's performance.
Same prompts as the rest of the field, blind human votes, ranked by quality and cost. It finished the 12 scenes at about $0.13 a scene. The other 4 scenes are still pending.
The only model that works for me is Astra, but it's expensive.
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
then again, I'm not a heavy AI user, so maybe I'm missing something.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
Almost no one is running DeepSeek V4.1 Flash on their laptop, so comparing based on API pricing seems perfectly reasonable to me.
At around 75% of the full model intelligence
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
It feels like that would be the expected result, all else equal.
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...
[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/
The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?
I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.
Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...
Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...
Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.
This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.
So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.
I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.
For a lot of people, there are other variables though.
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
Sooooo :D