Right now the "hosting" cost for inference is per-unit because it's new and expensive, but that won't last.
There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app.
The bigger problem for SaaS is that the floor has risen - people can build their own solutions for things that they used to buy SaaS for. So the industry needs to level up and solve harder problems.
I don't know about "very fast" or "soon" unless you're speaking in geological terms.
SOTA models like Kimi 3 require thousands of GB of RAM/VRAM to run at speeds that are real-time useful. Manufacturing the memory necessary for that quantity to be available at app-hosting prices will take decades. Software efficiency solutions might drop needed memory by an order of magnitude in that time...but a tenth of an enormous amount is still pretty darn big so won't get us there "soon".
It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
Not really.
>Every inference call costs money.
Not really, either. If you buy your own GPU, rack it, and run an open model, there is no unit cost. This is just expensive hosting infra. You also pay unit costs for SaaS that your software uses (things like SMS etc).
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
Are there any examples of products containing ai inference that are successful? Products that are beyond just direct access to frontier LLM's, I mean.
Products are only viable when you do a lot of work to solve a problem, which is then shared by many potential customers. It makes sense to amortize the high costs of solving the problem across all the different customers to reap economies of scale. Businesses then pay for product design, hosting, and maintenance because those costs can also be amortized, and they are cheaper than a bespoke solution for each customer. But if the bespoke solution becomes cheaper than that, because it's generated by an LLM that doesn't need to be paid a living wage, there's no reason to have the product in the first place. Just solve your damn problem and let other people solve theirs.
This is an underrated factor in the market structure of the AI bubble going on now. Anecdotally, we're not seeing new AI-based products other than the foundational models and coding assistants gain traction. Why? Because AI makes it so easy to customize the solution to your particular needs that everybody is just solving their particular needs directly. It's the opposite of the Internet boom, where the network greatly expanded the potential market, reduced the cost of reaching them, and made it economical to spend large amounts of money building a software product that had a TAM of billions. The AI boom instead enables extremely cheap customization, which shrinks the market to a single customer who uses AI to directly solve their problem rather than building a product that's generally applicable.
But, it's a hard problem. The models that run locally on normal computers/phones are pretty terrible compared to the frontier, without specialization and fine-tuning. And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
I think this is the critical point that would be interesting to see if it holds. Technology seemingly tends towards increased specialization.
The bitter lesson says the exact opposite.
Of course, and so does everything in the software world. The point is getting the cost so low that it’s basically free. The new DS V4 Flash or the smaller Qwen3.6 models are still really expensive compared to what we were used to in the economics of software, but it’s not unreasonable to expect these costs to continue falling down.
Rough chatgpt estimate says 3-5 orders of magnitude of difference compared to a typical user interaction with a SPA (db/cache lookup, CDN…)
Well, no. Copying is free, or so near free it makes zero sense to charge. LLMs are just papering over the damage caused by profit.
There are so many videos with hooks/teasers/'you won't believe what we discovered!!1', and now I just pause the video in the first second, ask "what's the tldr" and get the value from the video without a single ad impression (and likely racking up far more opex for Youtube than if I just streamed the video).
Most SaaS already works this way. M365 or Adobe Creative Cloud are great examples. They value it like a life insurance policy and find ways to make you sticky. It’s easier to just buy it.
The first round of AI products suck because they are not well defined. Copilot only makes sense if you do shit in office and SharePoint isn’t a dumpster fire. In my large O365 environment the bottom 50% of users use less storage than the top 2%. So why would i buy copilot for my janitor?
When M365 E9 reconciles invoices automatically with Excel, I’ll pay $150/mo and fire a bunch of people.
Absolutely not. Customers want systems for sales, reservations, accounting, and taking stock. That's where almost all the SaaS money is and none of it benefits from AI - and never will.
Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.