Hi, author here.

An important background is the imminent rise of actual LLM agents I discuss in the next post: https://vintagedata.org/blog/posts/designing-llm-agents

So answering to a few comments:

*The shift is coming relatively soon thanks to the latest RL breakthroughs (I really encourage to give a look at Will Brown talk). Anthropic and OpenAI are close to nail long multi-task sequences on specialized tasks.

*There are stronger incentives to specialize the model and gate them. They are especially more transformative on the industry side. Right now most of the actual "AI" market is still largely rule-based/ML. Generative AI was not robust enough but now these systems can get disrupted — not to mention many verticals with a big focus on complex yet formal tasks. I know large network engineering co are upscaling their own RL capacities right now.

*Open source AI is distanced so far due to lack of data/frameworks for large scale RL and tasks related data. Though we might see a democratization of verifiers, it will take time.

Several people from big labs reached out since then and confirmed that, despite the obvious uncertainties, this is relatively one point.

> So what is happening right now is just a lot of denial. The honeymoon period between model providers and wrappers is over. > In short the dilemma for most successful wrappers is simple: training or being trained on. What they are doing right now is both free market research for the big labs but, even, as all outputs is ultimately generated through model providers, free data design and generation.

This is a great observation on the current situation. Over the past few years, there's been a proliferation of AI wrappers in the SaaS space; however, because they're use proprietary models, they become entirely dependent on the model providers to continue to offer their solution, there's little to no barrier to entry to create a competing product, and they're providing free training data to the model providers. Instead, as the article suggests, SaaS builders should look into open source models (from places like Github, HuggingFace, or paperswithcode.com) or consider researching their own, and training custom models if they want to offer long-term services to their users.

> I've read a lot of misunderstandings about DeepResearch, which isn't helped by the multiplication of open and closed clones. OpenAI has not built a wrapper on top of O3.

It also doesn't help that they let you select a model varying from 4o-mini to o1-pro for the Deep Research task. But this confirms my suspicion that model selection is irrelevant for the Deep Research tasks and answering follow-up questions.

> Weirdly enough, while Claude 3.7 works perfectly in Claude Code, Cursor struggles with it and I've already seen several high end users cancelling their subscriptions as a result.

It's because Claude Code burns through tokens like there's no tomorrow, meanwhile Cursor attempts to carefully manage token usage and limit what's in context to remain profitable. It's gotten so bad that for any moderately complex task I switch to o1-pro or sonnet-3.7 in the Anthropic Console and max out the thinking tokens. They just released a "MAX" option but I can still tell its nerfed because it thinks for a few seconds whereas I can get up to 2 minutes of thinking via Anthropic Console.

Its abundantly clear that all these model providers are trying to pivot hard into productizing, which is ironic considering that the UX of all these model-as-a-product companies is so universally terrible. Deep Research is a major win, but OpenAI has plenty of fails: Plugins, Custom GPTs, Sora, Search (obsolete now?), Operator are maybe just okay for casual users - not at all a "product".

The argument appears to be "last-mile" specialisation of AI models by massive-compute-companies will be entirely proprietary, and walled-off to prevent competitor extraction of data. And these scenario/domain/task specific models will be the product sold by these companies.

This is plausible insofar as one can find a reason to suppose compute costs for this specialisation will remain very high, and the hardwork of producing relevant data will be done best by those same companies.

I think its equally plausible compute will come down enough, and innovations in "post-training re-training" will occur, that you'll be able to bring this in-house within the enterprise/org. Ie., that "ML/AI Engineer" teams will arise like SEng teams.

Or that there's a limit to statistical modelling over historical cases, that means specailisation is so exponentially demanding on historical case data production, that it cannot practically occur in places which would most benefit from it.

I think the latter is what will prevent the mega players in AI atm making "the model the product" -- at the level they can specialise (ie., given the amount of data needed), so can everyone else.

Perhaps these companies will transition into something SaaS-like, AI-Model-Specialisation-As-A-Service (ASS ASS) -- where they create bespoke models for orgs which can afford it.

> Inference cost are in free fall. The recent optimizations from DeepSeek means that all the available GPUs could cover a demand of 10k tokens per day from a frontier model for… the entire earth population. There is nowhere this level of demand. The economics of selling tokens does not work anymore for model providers: they have to move higher up in the value chain.

I've been using Cline so I can understand the pricing of these models and it's insane how much goes into input context + output. My most recent query on openrouter.ai was 26,098 input tokens -> 147 output tokens. I'm easily burning multiple dollars an hour. Without a doubt there is still demand for cheaper inference.

I don't disagree, but I think we need to be careful of our vocabulary around the word "model." People are starting to use it to refer to the whole "AI system", rather than the actual transformer model.

This article is talking about models that have been trained specifically for workflow orchestration and tool use. And that is an important development.

But the fundamental architectural pattern isn't different: You run the model in some kind of harness that recognizes tool use invocations, calls to the external tool/rag/codegen/whatever, then feeds the results back into the context window for additional processing.

Architecturally speaking, the harness is a separate thing from the language model. A model can be trained to use Anthropic's MCP, for example, but the capabilities of MCP are not "part" of the model.

A concrete example: A model can't read a webpage without a tool, just like a human can't read a webpage without a computer and web browser.

I just feel like it's important to make a logical distinction between a model and the agentic system using that model. Innovation in both areas is going to proceed along related but different paths.

> Inference cost are in free fall. The recent optimizations from DeepSeek means that all the available GPUs could cover a demand of 10k tokens per day from a frontier model for… the entire earth population. There is nowhere this level of demand. The economics of selling tokens does not work anymore for model providers: they have to move higher up in the value chain.

Even before DeepSeek, the prices were declining by about 90% per year when keeping performance constant. The way to think about economics is different I think. Think of it as any other industry that is on a learning curve like chips, batteries, solar panels, or you name it. The price in these industries keeps falling each year. The winners are the companies that can keep scaling up their production. Think TSMC for example. Nobody can produce high quality chips for a lower price than TSMC due to economies of scale. For instance, one PhD at the company can spend 4 years optimizing a tiny part of the process. But it’s worth it because if it makes the process 0.001% cheaper to run then the PhD paid itself back on the TSMC scale.

So the economics for selling tokens does work. The question is who can keep scaling up long enough so that the rest (has to) give up.

I'm confused by the language here; it seems "model" means different things.

To me a "model" is a static file containing numbers. In front of that file is an inference engine that receives input from a user, runs it through the "model" and outputs the result. That inference engine is a program (not a static file) that can be generic (can run any number of models of the same format, like llama.cpp) or specific/proprietary. This program usually offers an API. "Wrappers" talk to those APIs and therefore, don't do much (they're neither an inference engine, nor a model) -- their specialty is UI.

But in this post it seems the term "model" covers a kind of full package that goes from LLM to UI, including a specific, dedicated inference engine?

If so, the point of the article would be that, because inference is in the process of being commoditized, the industry is moving to vertical integration so as to protect itself and create unique value propositions.

Is this interpretation correct?

> This was the whole message behind the release of GPT-4.5: capacities are growing linearly while compute costs are on a geometric curve

I don't see where that is coming from. Capacities aren't really measureable in that way. Computers either can do something like PdD level mathematics research more or less under their own power or they cannot with a small period of ambiguity as subhuman becomes superhuman. This process seems to me to have been mostly binary with relatively clear tipping points that separate models that can't do something from models that can. That isn't easily mapped back to any sort of growth curve.

Regardless, we're in the stage of the boom where people are patenting clicking a button to purchase goods and services thinking that might be a tricky idea. It isn't clear yet what parts of the product are easy and standard and what parts are difficult and differentiators. People who talk in vague terms will turn out to be correct and specific predictions will be right or wrong at random. It is hard to stress how young all these practical models are. Stable diffusion was released in 2022, and ChatGPT is younger than that - almost yesterday years old; this stuff is early days magic.

Models could easily turn out to be a commodity.

I like this as a narrative framework a lot more than the AGI arc. This seems a lot more realistic and less dramatic, more human-focused.

I like the idea of a model being able to create and maintain a full codebase representing the app layer for model-based tools but in practical terms at work and on personal projects I still just don't see it. To get a model to write even a small-scale frontend only app I still have to make functions so atomic and test them to the point where it feels close to the time it would take to write the app manually. And if I ask a model to write larger functions or don't test them / edit them through 3-5 rounds of re-prompting I just end up with code debt that makes the project unrealistic to continue building out beyond a pretty limited MVP stage without going back line by line and basically rewriting the whole thing.

Anyway I'm no power user, curious what other people's experience is. Maybe I'm just using the wrong models.

Models might be the product, but data is the material that the products are made out of.

I'm starting to think that if you can control your data, you'll have somewhat of an edge. Which I think could lead to people being more protective of their data. Guess we'll move more and more in the direction of premium paid data streams, while making scraping as hard as possible.

At least in the more niche fields, that work with data that isn't very commonplace and out there for everyone to download.

Kind of sucks for the open source crowds, non-profits, etc. that rely on such data streams.

Not having API-s for models would suck terribly. It would kill tools like Aider or Cline, where I can switch models as I prefer, paying only for tokens I have used. The only option would be to purchase overpriced application from a model provider.

I hope the author is wrong and still there will be someone who would like to make money on "selling tokens" not end-to-end closed solutions. But indeed, market surely would seek for added value.

This is a thoughtful article, but I very much disagree with the author's conclusion. (I'm biased though: I'm a co-creator of OpenHands, fka OpenDevin [1])

To be a bit hyperbolic, this is like saying all SaaS companies are just "compute wrappers", and are dead because AWS and GCP can see all their data and do all the same things.

I like to say LLMs are like engines, and we're tasked with building a car. So much goes into crafting a safe, comfortable, efficient end-user experience, and all that sits outside the core competence of companies that are great at training LLMs.

And there are 1000s of different personas, use cases, and workflows to optimize for. This is not a winner-take-all space.

Furthermore, the models themselves are commoditizing quickly. They can be easily swapped out for one another, so apps built on top of LLMs aren't ever beholden to a single model provider.

I'm super excited to have an ecosystem with thousands of LLM-powered apps. We're already starting to see it materialize, and I'm psyched to be part of it.

[1] https://github.com/All-Hands-AI/OpenHands

These two trends seem somewhat contradictory:

> Generalist scaling is stalling. This was the whole message behind the release of GPT-4.5: capacities are growing linearly while compute costs are on a geometric curve. Even with all the efficiency gains in training and infrastructure of the past two years, OpenAI can't deploy this giant model with a remotely affordable pricing.

> Inference cost are in free fall. The recent optimizations from DeepSeek means that all the available GPUs could cover a demand of 10k tokens per day from a frontier model for… the entire earth population. There is nowhere this level of demand. The economics of selling tokens does not work anymore for model providers: they have to move higher up in the value chain.

Wouldn’t the market find a balance then, where the marginal utility of additional computation is aligned with customer value? That fix point could potentially be much higher than where are now in terms of compute.

> This is also an uncomfortable direction. All investors have been betting on the application layer. In the next stage of AI evolution, the application layer is likely to be the first to be automated and disrupted.

I'm not convinced, we tend to think in terms of problem-products(solutions), for example editing an image => photoshop, writing some document => word. I doubt that we are going to move to a "Any problem => model". That's what ChatGPT is experimenting with the "calendaring/notification". It breaks the concept that one brand solves one problem. The App store is a good example, there are millions of apps. I find it really hard that the "apps" can get inside the "model" and expect that the model will "generate an app tailored" for that problem at that moment, many new problems will emerge.

I am not so sure this makes sense. Training a model to directly use certain tools (web search etc) makes the model very specialized and less flexible. As long as other solutions are more flexible and less costly, training a specialized model would not be worth it.

I just saw a billboard sign for “deepseek on together.ai” on the 101 a couple days ago, and thought “that’s a very good idea”. As long as there are infra players like together.ai and truly open source crack research teams like DeepSeek and Meta (no one can monopolize research(ers)) I don’t think this “gated model” thesis holds. Which means that OpenAI and Anthropic are competing with every other token output wrapper, and somewhat poorly as they aren’t able to offer competing models where theirs may be lacking. Cursor defines this dilemma for the foundational providers.

Comparison I would make is that is like the transition from renting a server to using services on the cloud - you used to rent a server of a specific size and do whatever you want with it, but then the cloud came and undermined that "Swiss army knife" approach by having you rent specific services directly - database storage, processing, etc.

So AI will be more directed and opinionated, but also much easier to use for common tasks. And the "renting a server" option doesn't go away, just becomes less relevant for anyone in the middle of the bell curve.

> In short, what Claude aims to disrupt and replace the current workflows like this basic "agent" system from llama index: [Figure 1] With this: [Figure 2]

In the 2nd figure, I think we have a viable pattern if you consider "Human" to be part of "Environment". Hypothetically, if one of the available functions to the LLM is something like AskUserQuestion(), you can flip the conversation mode around and have the human serve as a helpful agent during the middle of their own request.

The real questions wrt the title is:

Will specialized models also hit a usefulness wall like general models do? (I believe so)

And

Will the model’s blindspots hurt more than the value a model creates? (Much more fuzzy and important)

If so, then even many specialized models will be a commodity and the application on top will still be the product end users will care about.

If not, then we’ll finally see the return on all this AI spending. Tho I think first movers would be at a disadvantage since they need much higher ROI to overcome the insane cost spent on training general models.

This is an incredibly valuable analysis.

In simple terms, performing a relatively simple RL on various tasks is what gives the models the emergent properties like DeepSeek managed to do with multi step reasoning.

The reasoning models and DeepSearch models are essentilly of the same class, but applied on different types of tasks.

The underlying assumption then is that these "specialized" models is the next step in the industry, as the general models will get outperformed (maybe).

Interesting read. I am curious how you would analyze the current AI hype MCP (model context protocol) from your perspective. Does it fit into the future you see? It seems like it's going the complete opposite direction as the future you paint, but perhaps it's just a stepping stone given the constraints of row.

> It's coding model no longer just generating code but managing an entire code base by themselves.

What model is the author talking about? I would pay for that. Is Claude really THIS good, that it could manage the codebase of say, PostgreSQL?

> an agent has to perform the targeted tasks internally: they "dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks".

> What most agent startups are currently building is not agents, it's workflows, that is "systems where LLMs and tools are orchestrated through predefined code paths." Workflows may still bring some value

While this viewpoint will likely prove correct in the long run, we are pretty far away from that. Most value in an Enterprise context over the next 3-5 years will come from embedding AI into existing workflows using orchestration techniques, not from fully autonomous agents doing everything end to end "internally".

> This is also an uncomfortable direction. All investors have been betting on the application layer. In the next stage of AI evolution, the application layer is likely to be the first to be automated and disrupted.

Highly agree with the sentiments expressed in this post, I wrote about something similar in my blog post on "Artificial General Software": https://www.markfayngersh.com/posts/artificial-general-softw...

I find the distinction you draw between weights and a program interesting - partially the idea that one is a “static file” and the other isn’t.

What makes a file non-static (dynamic?) other than +x?

Both are instructions about how to perform a computation. Both require other software/hardware/microcode to run. In general, the stack is tall!

Even so, I do agree that “a bunch of matrices” feels different to “a bunch of instructions” - although arguably the former may be closer in architecture to the greatest computing machine we know (the brain) than the latter.

</armchair>

So to clarify: the important product that people will ultimately want is the model. Obviously you need to design an infra/UI around it but that's not the core product.

The really important distinction is between workflow (what everyone use in applied LLM right now) and actual agents. LLM agents can take their own decision, browse online, use tools, etc. without direct supervision as they are directly trained for the task. They internalize all the features of LLM orchestration.

The expression ultimately comes from a slide from OpenAI from 2023 https://pbs.twimg.com/media/Gly1v0zXIAAGJFz?format=jpg&name=... — so in a way its a long held vision in big labs, just getting more accute now.

I wouldn't say it is correct. A model is not just a static file containing numbers. Those weights (numbers) you are talking about are absolutely meaningless without the architecture of the model.

The model is the inference engine, a model which can't do inference isn't a model.

Seeing LLM as a motor was a legitimate view until recently. But what we're start seeing with actual agentification is models taking the driver seat, making the call about search, tool use, API. Like DeepSearch, these models are likely to be gated, not even API accessible. It will be even more striking once we'll move to industry specific training — one of the best emerging example is models for network engineering.

The key thing really about my post: it's about the strategy model providers are going to apply in the next 1-2 years. Even the title is coming from an OpenAI slide. Any wrappers will have to operate under this environment.

I have seen this analogy before, and (hence the question). Apologies if it's rude. By my understanding, while the tools are important, most of the apps hit escape velocity as the underlying models became good enough. You had cursor doing decently well until Claude Sonnet 3.5 came along, and then it took off. As did windsurf. Perplexity and Genspark became 10x more effective with o3-mini and deepseek r1. Plus the switching costs are so low that people switch to apps with the most advanced model.(and ui is very similar to other app) Do you think there is space for apps which can keep improving without improvements to underlying models?

yeah, but engine/car analogy breaks down when it turns out all of the automotive engineering and customer driving data is fed to the engine so they can decide at any point to make your car or your car but better

> To be a bit hyperbolic, this is like saying all SaaS companies are just "compute wrappers", and are dead because AWS and GCP can see all their data and do all the same things.

isn't "we don't train on your data" one of - if not the - the primary enterprise guarantee one pays for when rolling out LLMs for engineers? i don't see a cloud analogy for that