Open-weight AI is having its Kubernetes moment(tobi.knaup.me) |
Open-weight AI is having its Kubernetes moment(tobi.knaup.me) |
So what open weight models do is at least provide a baseline of inference cost to add some sanity to the price markers. And of course predictability too--if you really want Kimi K2 instead of K3 you can still use it.
So the competitive pressure and predictability offered by open models is helpful for users
The price is what the market is willing to bear for the available compute capacity and competitive landscape. You can only discover that price after trying different price points and seeing what happens.
Everyone is trying different pricing schemes and discounts as they test the market. The demand is fluctuating at the same time.
It’s probably very confusing if you’re primarily familiar with stable and mature markets. Price fluctuations are a common feature of new and evolving markets.
I think this is a very important aspect, especially after the huge GPT-4o backlash when GTP-5 came out. Each model has certain quirks, and areas where the previous model might be better for some use cases than the latest and greatest, and the labs so far seem to have no desire to offer some kind of "LTS" release.
FlashAttention was a hell of a drug.
Artificial inflation to recoup R&D.
1. the field was nascent and new efficiencies were discovered
2. supply and demand
3. its in the company's incentive to make their models more efficient to increase overall usage so that while the margin remains the same, the total revenue + profit increases
I genuinely don't know what puzzles everyone?
LLMs are fundamentally tools intended to be useful. But LLM vendors don’t understand their systems well enough to actually price the product people are trying to buy (for example, the actual product of a coding model is the code that it produces, not the tokens, which are just an internal mechanical process involved in the creation of the code). Token based pricing is that lack of understanding leaking out of the organization that ought to be responsible for it, and being dropped on the user.
Imagine if we made cars like this! You’d go to the car dealer and ask for a car. They’d bring you a pile of parts, charge you for them, and try to put them together in front of you. You’d go back and forth for a bit, rephrase where you want the steering wheel, etc. Some of the parts wouldn’t fit but you’d be invited to pay for replacements as well. In the end you’d either have a car or not, that’s your problem.
If you're the best performing "computing cluster" (ie. whatever you call the entity that can complete a massive calculation), you get a blank check from Congress.
Why? Because you need those calculations to "pump" nuclear weapons. They are needed to calculate both the geometry to make fusion bombs possible at all and to calculate the effect of a given geometry. They are the reason US/Russia/China have the biggest and strongest weapons known to humanity. And of course, they were replicated worldwide for this reason. I mean not that anyone will admit this but we don't have the best possible solution, and we don't know either the upper or lower limits for fusion devices (plus the lower limit would be very useful for energy generation, which for the US would effectively mean almost literally unlimited large marine ships that never need refueling. And yes, the solution to that problem is almost literally a 3d shape. Not just that, but mostly)
Now it appears it does not work the same when you democratize computation. Humans want a particular amount of computation and are willing to pay a given price for that. But the accountants still saw the blank check from before and ... do what accountants do. Economics don't change because you make things bigger and accessible, do they? Oh ... wait a second ...
As someone put it recently though, we now have data. 2.3% of humans in the US are willing to pay $20 per month for the support of a model like GPT-5.5/Claude code. If that's true (and after years of having this model, why wouldn't it be?) ... it means AI startups are doomed (because it's not even 10% of what they need it to be to make economic sense).
We have a couple new nuclear weapon designs, but not really going for bigger or stronger. Just packaging.
We built the big powerful ones with 1960s computing.
Now, stockpile stewardship -- being sure that stuff will keep working without ongoing testing -- is a bit expensive in compute. You need early 2010s supercomputer power.
In other words, I strongly disagree that nuclear weapons are the primary driver of high-end compute.
But consumer uptake strikes me as the worst way to judge whether the big AI shops will make it. That's not where most of the leveraged user return or deployable capital is.
Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.
Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.
Quantised models running overnight go most of the way for non-coding tasks.
For the Hyperscalers...and Oracle...cant wait for the day...
Might make sense eventually. Same logic as companies working on Linux. "An AI model is a business necessity. But making an AI model is so expensive we should not make our own. So let's just use the open one, and contribute the stuff that we need."
Now, here is an idea that I have not heard before... and I think there is some merit to this. This is also the sort of thing that a state (looking at you CA, CO, IL, NY) could do, instead of just the federal government.
What is your stack (harness, model) and how much do you pay per month?
How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan?
I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?
oh now i see, the Chinese government is funding the training and release of their best models to pressure OpenAI, Anthropic, and others to do the same for competition's sake. I don't buy it, this seems more like a way to get SOTA models RL'd to comply with Chinese government approved information distribution. If I have to trust a black box of answers to questions i would trust one from a US for-profit publicly traded company subject to market forces over one approved, and heavily subsidized, by the Chinese government.
What's the incentive for the Chinese labs to continue releasing weights 5 years from now? It's not a stable equilibrium and cannot last.
- The lab spending large sums on research and training does not get the inference revenue to fund those efforts.
- Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions.
- OSS is often a two way street where features and integrations are built that the original author benefits from. Open weight models are largely a one way street because the marginal benefit is so much less than training costs.
In the short term, it means Chinese labs can attract talent and, I suspect, funding from their gov. Similar to every other industry the CCP subsidized to take over.
Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]?
N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid.
I still remember the time when Einstein@Home was in its heyday, and many people who loved putting together fast PCs contributed sometimes even for the reason of showing off in the statistics [3].
---
[1] https://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_f...
Once the weights are out, it makes no sense to ban them in the US while the rest of the world takes advantage of it. But that doesn't mean developers of frontier models shouldn't take steps to prevent their weights from being stolen.
Same situation in protectionism in AI. The rest of the world will simply lap us.
Another shill account I gues
While selling that LLM API usage, they then capture all the prompts, outputs, and intermediate thinking the LLM does, and then sell those logs to the companies making open-weight models. The open-weight model developers then train on those logs to 'distill' a model.
I can only assume this account is pure shilling for the closed-source AI labs.
Open Weights is free as in beer
We used to call that competition.
Imagine making this argument with a straight face in any other industry:
"The claim is that Japanese car companies are buying Ford vehicles, and then leasing them to American consumers at cut-rate prices. In return for the cheap cars, the customers are letting the Japanese observe their driving behavior, studying how they use their F-150 and then the Japanese car companies are applying that data to design new vehicles that will directly replace Ford!"
That's not what they are observing, it is the behavior of the engines.
The military has already mastered fusion (power), and likely has mastered gravity in some form in secret. They aren’t using mainstream compute for these discoveries
At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.
It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.
I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.
The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.
That said, even the foundational models fail at the hard parts of my code so I use it opportunistically.
I have reduced down to just using zed, will three locally hosted models.
Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps
Qwen 3.6 35B-A3B on 1x titan v + 2x1080ti llama.cpp with full context ~30tps
GPT-OSS 120b on pure cpu (slow)
I just use zeds parallel agents, task switching, stopping and fixing the code when a model gets stuck.
This still lets me stay engaged, and to modify code to be maintainable etc…
It gets me 80% there and I use to keep a subscription but often times just using googles AI mode is just as good.
That said I have 30 years of experience and insist on knowing how my code works, so this gets me 80% of the short term benefits while not depending on a 3rd party to keep my code moving forward.
Your mileage will vary and 2*5060ti 16gb cards would get around 100/tps with Qwen 3.6 35B-A3B on cards that are widely available.
To be honest the more modern cloud models are using draft tokens etc… that while they are superior for common coding tasks are degrading with more domain specific tasks.
That is just the cost of the draft model being ~10-20% of the foundation models size, and even the biggest Blackwell GPU is limited to ~250/tps so MoE or draft models are required for scaling performance at the foundational level IMHO.
The hard part is my use case are the OOD or small examples in corpus level, the above hurts there.
A Lamborghini may be nice, but I personally need a minivan more.
Can't comment on how it compares to plans (I really don't like the limitations and general shenanigans I see around plans, so I've never tried them).
It is notably slower than Fable / Opus / Gemini, but also vastly cheaper than their API pricing.
Deepseek is at least on par with Sonnet (ghcopilot at work)- I don’t use opus, too spendy and I don’t need that level of ability.
The cost is for me was $5/6 months of use. Not a big user I guess! It’s good though, fast enough and incredibly inexpensive.
Been testing Qwen 3.6 28B on a 5090, and it’s also quite good for “free”.
I mostly do small self serving embedded projects based on esp32, so not very complex.
I think that's the case for people who compare it to proprietary models paid via API—which I think is irrelevant given the majority of people daily driving AI coding are doing on a subscription plan.
The better analysis then is not about AI coding, since there's no subscription plan for Kimi K3.
Instead, compare the cost of running some agentic _task_ that isn't coding which can only be done via API. Think of all the startups wrapping around ChatGPT and Claude to provide some additional set of tools, context, data and hoping to turn it into a profitable service.
To those companies, which are many, open models are the difference between the math working out today vs. praygeing they can scale fast enough to find profitability.
When I expect to need a lot of tokens and the task isn't too difficult I use sota to plan and create a thorough set of instructions and let deepseek chip away at it. With thorough instructions the quality tends to be satisfactory, and you pay something silly like $15 for 600m tokens.
GLM 5.2 seems like a decent price/perf and Kimi 3 has some real nice performance for an open weights model, but gpt 5.6 is unexpectedly affordable (especially if you don't automatically use Sol at max) so I don't think either is worth it atm. The exception is when you're working on something that US models get cold feet about, which seems like a constantly growing list. For me Fable is already too much of a headache in this regard, but chatgpt is still okay-ish. Hopefully it'll last. If not, there's Kimi.
tldr SOTA for most things because gpt 5.6 is token efficient. If I expect to burn a lot of tokens I use deepseek 4.
It's not as good as the frontier models I use at work, but it's plenty capable for the types of tasks I am using it for.
There is something special about complex systems that just-work(tm)
If you think of docker as "kinda like vms except not really" and k8s as "kinda like deploying and composing docker containers but not really", this may be for you:
https://ojensen.net/infra/understanding-k8s-1
It's actually really neat, i wish i had bothered to learn it years ago.
Its 2026 and its the de facto method of deploying software basically everywhere.
You gotta really work for it to not know what its for by now.
We walked in, and it was fine. Because it was all kubernetes and laid out like every other app for the most part.
The kube hate is just sad at this point. You need to know like 15 concepts that are all applied in the same way. It mostly just works.
What 15 concepts? You're making me worry that I missed something. It was straight forward: pods, nodes, hw type, lifecycle, deployment. They run almost the same docker as the old ec2s used.
What did I miss? Is there something important I need to read?
I’ve been running my own personal k8s cluster on digital ocean for the last 5+ years now and it’s dead simple.
Takes me 30 minutes to create a new namespace a deploy an app, love the bonus of having complete flexibility on my stack too — PVC + SQLite ftw
The value built on that stability was probably worth acquiring, and it’s infra will decay.
I do think that corporate price gouging is a huge problem that does need to be addressed. But especially with smaller business types — creatives, et al— I still haven’t gotten any grownup answers about what would compel people to get professionally good at something and innovate in the complete absence of copyright: the vastly better business model would be waiting for someone else to do something new and interesting, stealing their work, and then undercutting them in the market because you don’t have R&D/et al costs to recoup. You can’t say that wouldn’t happen because it’s exactly what the AI companies did to billions of people, scoffing at any protest. And ironically, they’re now whining about the Chinese doing it to them.
But yes, the reason brand name drugs are drugs are more expensive than generics is due to intellectual property, both the patent and the trademark.
From my very ignorant standpoint, K8s seems to be about running a "cluster", but I don't know why I would want to do that.
This isn’t as unusual as some people are trying to make it sound. This has been happening since the dawn of finance.
I thought this would be less foreign to everyone since we just went through this whole conversation for a decade with Uber and Lyft. Their demise was predicted from the start from everyone who thought that it was going to collapse as soon as they couldn’t subsidize your rides with promos. There was much wailing and gnashing of teeth as their prices changed to feel out the market. Then they found profitability and the critics went silent.
That is some really impressive bubble you are living within. Basically everywhere - nowhere near that, no.
[edit]: Maybe containers, but software in general is so much more broad than containers.
I'll agree that (nearly) everyone should know when Kubernetes is useful, but let's not pretend its the default method for everything. Even then, choosing to deploy on K8s falls on the sysadmins/DevOps I wouldn't expect the devs know or do much more than provide the Dockerfile.