When AI Costs More Than the Engineer(tomtunguz.com) |
When AI Costs More Than the Engineer(tomtunguz.com) |
Though I agree it might be informative to split it by industry sector.
Compare AI costs per-engineer-salary-dollar, because more expensive engineers probably need more expensive AI.
You should, but with two important caveats. First, you don't know what their amortization schedule is like so you don't know what the impact on the pricing will be (are they going to pass the cost on over 5 years or over 20 years?), and second they may go bust before paying the cost down so they may not get a chance to pass it all on. If someone buys the company then they'll get a discount on the value, which means the training costs are just eaten by the investors.
Anthropic spends [...] about $2m of compute per employee per year against a likely all-in comp of $500k+.
The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI
This framing makes no sense. The reason Anthropic spends so much on compute per employee is that they are building models. Anthropic employees aren't opening Claude Code and spending $2m in inference every year, so comparing it to other software companies, where AI expense is mostly inference, is completely incoherent.
Yes, the cost has to be passed down eventually, but it's not passed down to one company; it's passed down to all of Anthropic's customers, so the actual share of that money will be distributed among Anthropic's clients.
Look, I 100% agree with the idea that OpenAI and Anthropic are both unsustainable companies that have dug themselves so far into a debt hole that, most likely, the only way they'll be rescued is with government intervention, but this is still a terrible article.
Bandwidth isn't free, and all my life I've been told that piracy is theft.
Assuming their investors win the bet they placed on them. Which isn't given.
If you want to take the DDG LLM summary at fate value, apples are lower in calories and sugar but higher in fiber compared to potatoes, which are richer in vitamins and minerals like potassium and vitamin B6. Overall, apples provide more dietary fiber, while potatoes offer more protein and essential nutrients.
Comparison rarely lead to one obvious all superior option that discard every other considerations.
Evian use 1.25 million litres of water per employee per year. When can we expect other non-bottled-water corporations to rise to this level of water usage?
I have developed some intuition of how large tasks I can give it so that it will complete them well, probably erring on the conservative side.
I am using it daily for all my code writing and honestly don't remember the last time I had the feeling that I had to spend a lot of work to get the last few % done.
Lower prices will not reduce AI spend. They will simply increase usage.
There is no real ceiling on how much companies can delegate to AI. The only limit is the floor where spend too little, and you simply stop being competitive.
It is unlikely this kind of agentic workflow will ever get cheaper. Agents get stuck in doom-loops quite often, just burning tokens without any value. Especially by prompts created by people unfamiliar with the codebase.
And it is becoming increasingly obvious that better models just use more tokens (and take longer to execute on prompts). So this kind of human-out-of-the-loop workflows will be forced to use cheaper models and be time-gated in order to not waste tokens. And then they will also produce worse results than a manual change or a more powerful model...
If tokens get cheaper you just put a better model for this kind of problem and let it run for longer.
But what is more insane is that we are using a ton of cloud VM time on top of a ton of tokens just to save a few minutes from a developer doing the same on his machine...
I don't think my company will keep this system once the free credits run out once they realize how much it actually costs.
For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.
Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.
The problem is going to become that there's no incentive for anyone to run the stupidly-expensive training phase. May God have mercy on the stock market.
How many software developers were working on code like the one you describe?
So, overall, you get more done that without AI, at the cost of spending almost all of your time writing specs and doing code review and almost none of it writing code.
Do you get 3.3x the work done? Probably not. Do you get 2x the work done? I think maybe, if you can hack the dynamics of the new job as a manager of eager robots. For me the jury's still out on the second point.
It is really crazy people didn't think this through.
Excuses for these exercises will vary, AI is just the latest; but it's fundamentally just a labor-containment/efficiency-seeking strategy.
They were lucky with the ad empire he built and that‘s it.
Unfortunately they also don't realize just how much decision-making real people do lower down the org-chart. Critical decisions are often done by the leaf nodes, often without even discussing it internally with the leaf-node team. AI will likely not be very good at this kind of decision making or realize any decision needs to be made at all.
We don't get unlimited hiring budget, so we also won't get unlimited token budgets, and we as the operators will be responsible for the productivity of our agents.
What does performance management for engineers look like when dollar token cost is included in reviews? I think it's going to change a lot of assumptions and a lot of strategy around AI use.
> The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI, 40% of a fully-loaded $224k senior engineer salary. The median spends $137. That is the gap : ... 0.4x at the top of the market, near zero at the median.
So it's not more expensive than an engineer it's 40% as expensive, and for many companies use-cases the cost is virtually negligible.
Even here in Europe where developers are much cheaper than in the US, it still makes sense to pay for the LLM Enterprise subscriptions.
Does it though? I do not see any advantages in my day to day job over using the cheaper models.
But you can get an awful lot done even with just like $200 a month at API pricing if you are careful not to waste a powerful model on an easy task, or carry around a bloated context window etc.
I think a lot of the 'tokenmaxxing' people spending thousands every month are simply using the tools ineffectively (like having loads of Opus agents doing tasks that Sonnet or even Haiku could do). I suspect this will only get worse now with the release of Fable, but Anthropic must love it.
When you say the cheaper models do you mean like Deepseek or GLM? I haven't tried those but they look interesting. It'd be nice to shift to open weights and not be tied to one company.
This is almost economics level of line projection.
It would be good to understand _why_ anthropics "AI" bill is so high. First, They are going to be renting a lot of inference compute just to service customers (Meta's Capex bill is about 2x its wage bill) It then also needs a huge amount of infra to both run training and experimentation. THats probably a third of the cost. (storage and physical infra to get the most out of storage and compute is hard. Then getting it reliable, so that shit state doesn't propogate across the shared memory plane is very hard.
The other thing to note is that claude usage inside anthropic is tiny compared to the customer's usage. even with uber agents at "mythos++" its going to be at best a few thousand servers. not like the massive fleet needed to serve the paying customer.
So using anthropic as some sort of rational target to base any kind of prediction is madness. Its like looking at lyons tea rooms and going yeah, every company is going to spin up an R&D arm to make a company specific computer: https://www.sciencemuseum.org.uk/objects-and-stories/meet-le...
ALSO this assumes that the current way of running LLMs is the way forward. Custom software is expensive (in both time and tokens) to look after, its much easier and cheaper to buy it in from SaaS companies and let them figure that shit out. (yes I know SaaS apocalypse, but you are paying for real world experience, and a packaged way of doing things, rather than experimenting your self, where in a lot of cases the company doing the experimentation doesn't know what its doing)
This is not even specific to capitalism or VC mind you. Look how PRC led to the Great Chinese Famine. That’s why actual democracies (not the inter-elected aristocraties ), despite all their downsides, are so damn interesting. Corruption, negligence, or mere error with catastrophic follows, is easily spread in a situation where small core of individuals monopolize greatest part of decision weight, but is logistically impossible to achieve in a system optimized for widespread and highly redundant power responsibilities.
> The rest of the software market trails.
This shows how VC firms see things and why we have such a lopsided market where grift rises to top easily.
Yes the rest of the software market trails in comparision to the compute costs at Anthropic if you including training the actual models. Like is this the insight? Biggest AI company spends a lot of money to make AI models?
Sure you can find anthropic's business model risky/not feseable but using this as your starting point shows a lack of basic understanding at best and malicious intent to make a stupid point at worst
Let's see how this works out in the long run. For a historical analog, more expensive engineers don't use more expensive computers (by and large).
I often wonder what kinda features other devs implement compared to me, if they need that many tokens?
It kind of feels impractical to bloat up an app with features one barely understands? I've just been reading about these devs using x-amount of tokens, having that y-amount of steps perfected AI workflow, but none of them ever talk about what they actually implement all day...
Big AI labs are not software companies where payroll dominates expenses. They're capex-heavy industrial entities; it just so happens that the "machines" (whose output they sell) are nominally the same category as the devices that their knowledge worker employees use on their desks.
Anthropic was profitable last quarter.
If Anthropic can block distillations somehow (which are fair game imo given that Anthropic et al did the same with the written works of mankind), then they might stop or slow down the chinese from catching up.
Chinese also have like 40% of the AI researchers of the world, plus they have access to a lot of cheap labour for writing training data. I'm sure an hour of training data creation from one of China's 162 million university educated people is much cheaper than an hour of work from one of US's 97 million. Probably still cheaper than someone from the grand area.
China is behind in AI chips/GPUs but they are catching up. One thing where they have a hard dependence on outside is their energy imports: they have to import a lot of stuff from third party countries. The US on the other hand is energy self sufficient.