SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence(cognition.com) |
SWE-1.7 Reach Near GPT 5.5 and Opus Intelligence(cognition.com) |
Apparently 'free' on the $20/mo Devin plan (presumably within some quota still)
and that is "via Cerebras at 1000 TPS" according to the announcement
I live on Opus 4.8 High and their benchmark scores SWE-1.7 slightly higher ... if at all realistic that sounds like a great deal ... too good to be true?
What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.
I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.
I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.
They did not.
Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for.
Of course not having to pay the premium, because the routing is smart or whatever, would work just the same for me. I just would not want to have to think about it.
I'm an OpenCode user, but I'll fall back to Claude Code if I want to use Opus end to end for something, given my company has a subscription. But I'm not using yet another tool and subscription for a model that isn't even winning.
This whole company seems to optimize for raising money and impressing VCs. Lying about their products, ignoring consumer market to target enterprise, bragging about how they work their employees like slaves, and writing these posts full of intimidating technical jargon...
Remember when AGI was going to replace all jobs in 6 months? It's always been like that.
Could you expand on this?
Imagine how far community might have pushed if 2 past versions of 'morally superior' Anthropic and 'completely Open AI' open sourced their models for the community to build on top of them
As a (former) Windsurf user I'm pretty happy with the progress of the Cognition/Devin ecosystem after they took over Windsurf, now known as Devin Desktop.
* Based on the first comment in the link that claims to summarize the video.
For hobby projects I've completely switched to DeepSeek v4 pro. I spend less than on a $10 Claude plan and am not subjected to quota limits (when I have time and motivation, the last thing I want is a 5 hour quota running out). And the difference in model performance is fine for those smaller projects, most of which will end up abandoned or in a state of "good enough" anyways
And for utility tasks, those 30b models are also great. I'm a big fan of gemma4
At least with low level programming languages. They're all very good for webdev stuff.
Should as in "would it be nice?" - yeah. Should as in they have to? No.
> Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so
You can do pretty much anything you want with an MIT license.
I remember them saying a few years ago that, they didn't think it was worth specializing models for code, because their general purpose models kept beating them. I guess they changed their mind? Since they did start making codex models again.