Is the industry ready for tokens-constrained work?(blog.alaindichiappari.dev) |
Is the industry ready for tokens-constrained work?(blog.alaindichiappari.dev) |
I’m not an AI-maximalist, but “work a few days with AI and the rest of the month without because of cost” sounds literally crazy to me. (If you think AI is low/zero/negative net value, don’t do the first half; if it has net value, don’t do the second half.)
"Too much of a good thing" does exist. Someone might feel that it takes time for their organization to "metabolize" the rapid additions made by coding agents. That seems quite reasonable to me. Your comment seems to argue a false dichotomy.
Finance teams above all else value predictability. Token spend is the opposite of predictable. Eventually these two forces are going to collide and some part of the system will break. My guess is it's token spend, not finance allowing unbound spend on a previously predictable part of the balance sheet.
If you are the kind of person who thinks AI will drive down the wages of software engineers that much, you have to think that it is at least a 2x multiplier.
$65k salary all in for a company is easily $100k. So spending anything less than $100k would be what you would aim for.
Spending $10k and then balking at an additional $2k can make sense to me based on the circumstances. It's an additional 20% that hasn't been budgeted for.
The question is partly about what that 20% gives the employer - for some businesses, you might see diminishing returns from additional productivity of a software developer. E.g. I did some work creating BI reports and warehousing for a manufacturing company. Being more productive just meant faster reports or more reports. But faster/more reports didn't necessarily equate to more money for the business.
Sometimes it's a question of a business being penny wise and pound foolish, but sometimes it does make sense to me to not spend this additional money.
No idea, but companies refusing to buy a 2000$ laptop once every 3 years, a new mouse, keyboard, or screen for < 500$ are very common, more common than the other type.
So I'd expect those to not suddenly change into "let's waste money" companies.
We're 15 days into this new policy and its going ~okay~. Engineers adapt as they do, and have been leaning on `gpt-5.6-luna xhigh`. Some contractors have already hit their budget limit for the month.
A couple observations here:
1. Because LLMs/agents are tools, limiting their usage becomes a distraction and ends up being more of a drag on each individual's productivity. Instead of "just doing the work" engineers are now wasting time tinkering with setups (graphs, caveman skills, etc.).
2. Skill atrophy is real. Engineers that hit their budgets are seemingly less productive and less capable which is deeply concerning.
3. There are legitimate conversations happening now to explore open source harnesses (opencode/pi) and open weight models at the company in order offset the costs associated with going through a standard provider.
4. Token prices are venture capital subsidies. Its essentially free samples to get the market hooked on their addictive white collar drug. Remember when an Uber cost $7? As soon as OpenAI and Anthropic go public, they will need to begin showing progress toward profitability. That is when the true price of a token will be revealed.
if i’m not mistaken, in the past, people had limited time on “the mainframe”. so eventually they brought in what would be the equivalence of a local model… small computers that could do smaller tasks locally so they didn’t have to keep shelling out money to the mainframe gods and be handcuffed for usable time.
i could absolutely be mistaken that this is how it worked, it was before my time. but this is how people explain it worked for them.
i don’t think most work gives a shit about soa. smaller repeatable tasks can absolutely be run just fine on smaller local models. sure, we’ll upgrade models occasionally just like we went from suitcase sized laptops to whatever we use today.
this idea the hypedorks are pushing that soa is the only way is hilarious. hobbyists spend stupid money on classic cars, tools for woodworking, or whatever. spending money to do a hobby at home has never stopped wonks and their hobbies and businesses will do the same, spend to run models locally.
the sooner the hypeTrash does what hype always does, fades to irrelevance, the sooner we can get back to work.
You know what happened when a PC broke? You know what happens at a construction site when the excavator breaks down?
Exactly nothing — and that's okay.
It’s an artificial constraint. I could afford to spend more on my hobbyist programming but I choose not to.
I suspect that I’m adapting to the model by giving it more concrete guidance and guardrails. I read code more and I tell it to refactor code that I don’t like. Working within constraints isn’t all bad.
I’ve learned how to fully embrace agentic tooling to do tasks like keeping up with vulnerability reports, initially triaging defects that come in, etc. It would be hard for me to do things the old way at this point when I know tools are available that could make me more productive for a particular category of tasks.
My employer is considered capping all engineers at $200 or $500/mo of token spend depending on level. I regularly spend over $1k/mo today, but believe I can make a strong business justification for the value those tokens are creating.
At this point, I think engineers may be asking what token budgets are when considering new roles.
Tomorrow 2 engineers at 50% token time will change to 1 engineer with 100% token time for the same work.. the latter is too cost effective.
Right now, LLMs are the worst they'll ever be, but imagine what they will be like at the peak: Anything you want to code, coded instantly. Not waiting for code to stream in or wait hours for some agents to crunch through loops and planned: it just appears on the screen instantly like the way a webpage loads. Then imagine it can be done locally, on your device. Need an entire new custom operating system from scratch for some obscure hardware? Done. Here it is.
That's going to be like pure crack to anyone who needs to do absolutely anything. That's going to be like our generation's version of "today's supercomputers will someday be in everyone's pocket".
Generic CRUD/Config stuff I do for work (and assume most software jobs entail? Maybe not correct) doesn't use a significant amount of tokens.
In addition I have virtually unconstrained usage at work (for now).
Tasks that I do on my personal account: - building stuff for my wife's business - analyzing and experimentation and building harnesses to test other models/services - processing interesting research papers that publish without code or sufficient data to replicate the work
And that's besides simple stuff like exploring new topics I am interested in, and building learning assistants and personal tooling, and eliminating technical chores.
I almost certainly wasted tokens, but right now things are subsidized, so it works.
I don’t run into many issue with my personal subscription plans. However, at work while companies like Anthropic offer “Enterprise” plans, you still get billed at API rates which are quite high. The subscription plans are an incredible deal and subsidize these tokens, but these plans aren’t offered to large organizations.
What if they feel like they are getting value from it, but are actually not?
Claiming you have no idea why the managers had to do this felt to me like avoiding any responsibility
Our policy is quite permissive for exactly the reasons I conveyed above. It takes an extremely finely-tuned sense of value and high assurance that you’re on the diminishing return portion of the curve to conclude “SWEs should have this precisely metered amount (rather than zero or ‘as much as they don’t waste’)”
My claiming to have no idea was not an abdication of responsibility but rather a claim that it was likely an error (assuming a non-trivial team size and a business that can continue to grow).
10k might be a bit too high, but it's far from "absolutely ridiculous" amounts of being too high. An employee with a 5k salary can easily cost the employer 7-8k
If you have huge salaries, yeah sure 500 dollars its okay even if you only get 5 10% more productive.