I think I have LLM burnout(alecscollon.com) |
I think I have LLM burnout(alecscollon.com) |
https://github.com/JuliusBrussee/caveman
It's for getting it to output shorter answers, but also could help with your burnout.
Review AI code line by line is like watch movies frame by frame, and is impossible, very difficult, terribly boring, or abandoned sooner or later.
A good coworker will admit not knowing something, or if unsure give their best guess but discuss its limitations and why they might be wrong.
Question: Has anyone experimented with using voice to directly prompt an LLM, without doing speech-to-text? If an LLM can pick up on the skeptical nuances in a person's response, it might be prompted not to be overconfident in its subsequent output.
Getting sent IM responses that are copy pasted LLM nonsense. Getting a massive PR to review that was generated overnight and the author didn't read it first.
https://en.wikipedia.org/wiki/Ironies_of_Automation
https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...
Of course if you're supposed to achieve so much output that it's not possible to do anything but vibe it, fair enough.
I do not understand these complaints. Yes, those are the defaults and they're annoying, although the general public seems to like them. But you are not stuck with these. You can just tell the LLM how it should interact with you. If you're using any sort of harness beyond the chat window in a web browser, you can codify these instructions in a rules.md file or similar and have it automatically included in any new chat. It's not any harder than changing the default wallpaper or color scheme on your desktop operating system.
In reverse order, you can just tell the LLM to never use emojis. I don't like emphatic staccato fragments either, so I tell it to eschew the language of marketing and hype and stick to a factual and plain language, or to employ an academic tone. I explicitly instruct mine to ask clarifying questions whenever context is ambiguous and to push back on false assumptions or common misconceptions (by me). Hallucinationsa re the biggest problem of those you mention; it's not easy to totally eliminate them (for the same reason it's not easy to instruct people to not fall for scams or disinformation), but you can considerably reduce them by setting standards for citations.
I have ideas about reducing hallucinations over work material (ie a codebase) but am omitting them here as they are not fully thought out or tested.
Do does managnig agents.
What helped was a sleep and work system, oriented around being offline that was inspired by nature and from my earlier days in working in tech while car camping across the national parks.
Basically: the sun wins in terms of how all energy on the earth is structured, and expressed. All manners of cycles of organisms and living systems are in relation to its rise and fall, and even its particular color spectrum phases (whether thats night oriented or day). I call this our real circadian rhythm; it's used to being signaled by the light of the sun and maybe fire for millions of years and it isn't until recent centuries when we started tricking our biology with LEDs and lights. So the solution is simple. Orient yourself around the light of the sun and make sure it's the first and last major light source you see; blue limiting is the most important part BEFORE sunrise and AFTER CUT OFF ALL BLUE LIGHT. On my Mac I use a red light filter (using it now, it's 11:07pm ET and the sun went down about 2.5 hours ago). It's really hard to stay alert and chatting with an LLM when the only light sources are red and you keep them dim at that. Our ancestors would rest when the sun's at its peak (~1:05 pm today) and that's a good time to divide my own day productively as well. With intentional breaks diving the middle of the day with sunlight anchoring it, my nervous system is more relaxed, and by the evening time, it's also ready to transition out of anything blue-light assisted and most intellectual work and problem solving falls into this bucket. It's really hard to explain but it really works so simply. To enjoy the process a little more I made this fun sun clock, check it out at https://sunsignal.app
Cured my lifelong “night owl” “trait” in a couple days. Shockingly effective.
Turned out to be hard to keep up and still, like, exist with other people, and you’d probably need to relax it a little in Winter unless your job lets you work reduced hours to kinda “hibernate” (otherwise when would you do anything that’s not work but requires light or electronics?) but it sure worked.
At work, I ended up doing other chores, getting a lot more involved in projects I wouldn't even care to touch. Turns out it's kinda fun being the source of truth at work. I now have a clear sketch of what the company has done and what we can improve on.
Being able to fill in the gaps at a company that doesn't do much feels like a company within a company. Sure, its not Silicon valley, but it's still fun! And job security is guaranteed if you do a bit more than just play the ticket factory
I am happy about all the little side-projects, and ideas it help my realize, and I enjoy exploring this new world, but I've noticed LLMs feed my unhealthy "don't want to take a break and waste time being idle" mindset, and I need to correct it.
W.r.t. article's main complain - I think the similar thing happened due to factory manufacturing automation. What used to be a varied skillful craft in a shop became standing in a single place of an assembly line doing the exact same thing whole day. LLM took away the more creative and variable part of the work, and left the repetitive QA rubber-stamping. Probably some of the mitigations used back then could be rediscovered today.
I had to think of the factory scenes in Charlie Chaplin's Modern Times. The author's feeling is basically the main idea of the sketches, i.e. humans having to follow the pace of the machines instead of the other way around.
Reverse centaurs are nothing new. Ask any worker movement from the last centuries.
I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users, project docs, business plan docs. A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff. They're able to generate this stuff way faster than we can review. It used to be that it would take a significant part of a day for a project manager to come up with a planning doc - now they can generate one in a few minutes and send it out for review. It's just really tiring.
Getting an LLM to vomit out a bunch of documents and sending them straight to another colleague is absolutely unacceptable behaviour.
I think we will very soon move to a prove to me you've read it protocol and/or introduce speed bumps to slow things down.
He who brings the slop cannon shall be drowned by slop rain.
That ratio has changed, and verification is the hard part.
Verification is the point of all markets (and a decent part of human civ as well).
And review isn’t cost less - https://en.wikipedia.org/wiki/Ironies_of_Automation
I see a different type of pressure: I'm at a company that still is requiring everyone use LLMs with token leaderboards, time-spent measurements, and impacts to performance reviews, and all that. So I find myself having to carve out some percent of my time to stop doing productive work, and "go do AI to show token use." So my workload hasn't changed (or it's gone up), but I have N% less time to work on it because I have to spend time appeasing the AI gods...
I think this is in part because I am one of the software engineers that always liked building products more than writing complex software. So, I am driven by the feeling of creating something. And I want to get the feature perfect and complete. But getting from 95%->100% done can take a long time with UI work for me.
So I work much longer hours now, unfortunately.
But it's probably a common feeling. I wonder if we'll see an increased number of people burn out in the serious, medical sense.
Same, but I really have to fight the urge to just add fun new features to things I work on any time inspiration strikes. I am an appalling 'feature factory' if I don't actively keep myself in check. The cost of just building everything is so low, but the value of those things is also incredibly low, so I'm often just bloating what I build.
There's been a lot of articles and posts about the increasing importance of 'taste' in software built with AI, and I'm finding I know need to look for strategies to find some.
Spoken like someone who is not at an org/team that has undergone layoffs and reduced hiring in the last 3 years.
You might be in the minority there - especially when it comes to those who are facing burnout.
Cost of generation has been reduced, and is highly subsidized currently.
Cost of verification has effectively not changed. I’d say as a rule of thumb: verification is the tough part.
Our brains don’t fare well under constant review pressure. https://en.wikipedia.org/wiki/Ironies_of_Automation
"I did a Chat output, please fix and review it " is the kind of thing that empowers the people who used to have a minimal productivity, and now lets them to wreck things on an industrial scale.
It's not. There is no one person that has universally good taste. Also, we're not in your head, no matter how much better of a coder or whatever. We're not in your head and it's all terribly painful to navigate.
AI is not a productivity multiplier. There are diminishing results.
The ones that notice the highest increases of productivity are usually the ones that were unproductive at best and dangerously incompetent at worst.
Lots of companies (nearly all, I’d wager) of any size were leaving bare-minimum a 2x software development speed increase on the table before LLMs, having nothing whatsoever to do with how fast anyone was typing or thinking up code, and everything to do with how they organized and supported development work, and with your basic ordinary corporate dysfunction.
My company, I’d say it was more like 4x or 5x they could have achieved before LLMs, by fixing processes and reducing how often management steps on their own dicks.
All the people I’m seeing with crazy-high LLM productivity at my company? They’ve been given enormous autonomy to basically go do WTF ever they want, and people are jumping to get them anything they need (and most of what they’re doing is prototyping, for that matter). So right off the bat, if they’re competent, they should see a notable multiplier on productivity even if they weren’t using LLMs. Not that those aren’t helping, too, but if you don’t change processes they’re not all that effective, because the problem wasn’t speed of code-writing (and if you can change processes, you already could have sped up development a lot before LLMs…)
I confess that the above variant on the quotation is how I originally read it. And that's just about how I feel now with trying to sort through vibe-coded slop projects that are put forth by (well-meaning, probably good intentioned, not evil) people who represent them as if they're the handcrafted result of one dedicated developer.
That said, your reaction is totally human. I personally get sick of how the LLM writes prose with always the same tricks and formulas (even if you prompt it not to). Humans need variants and novelty, that's why fashion exists. We get fed up with repetition and after seeing too much green shoes, seeing a red one is so relieving :) (quick note: I don't like fashion - I'd advocate diversity and personal styles, not fashion)
But that's also the way you work with AI that might be part of the problem. Personally, I don't review all the code the AI generates. I look at it, and I review only the code that matters. And with time on a given project I review less and less because I trust more the architecture and ability of the AI to follow it. In my settings, the AI gets confined to the existing architecture (that we define together at the beginning of the project), and has to ask for authorization to create new things (that's when I review the more). Hoping this could help to avoid burnout myself...
I’m building personal projects at a prodigious pace. In a role reversal I treat the agents like I’m one of my clients (albeit a more technical one who gives them architectural direction) and they are me. I’m using the apps and tooling they make every day. I’ve cancelled SaaS subs for tools I’ve built myself.
I watch the tool calls and realize I should be better at core command line tools so I have a study plan to catch up (just a little bit a day). I’m revisiting long standing config that I dropped in to vim and tmux way back when I started and didn’t know anything.
I guess in theory I could hold my productivity to previous levels and read more. But it doesn’t feel like that’s possible. It feels like we are in one of those sea changes where the promise is less work, but the reality is increased productivity and expectations (the Industrial Revolution feels like the right parallel to reach for). Increased expectations happen in small ways and large. The agents are so good at polishing data presentations that I always send cleaned up visually impactful reports that would have taken significant time in the past just as a matter of course now.
But, I’m tired. I’ve spent the Fable on subscription window sprinting through as much work as I can before it goes API only. (As an aside, I don’t understand how everyone is using so many tokens. I’m sleeping very little and running as much code as I can through fable and I can barely touch a 20x max plan limit.) I keep telling myself I will slow down when it comes off, now it’s extended to the 12th and my window just reset, a few more days to keep knocking out backlog items. I feel like I have to keep the robots busy overnight so when I wake up I can immediately sit down to review. I give directions to agents on my phone which feels wild to me.
Prior to the last 12mos AI companies were hell bent on squeezing out the best results from mediocre models.
But... now that the top models have progressed, those same AI companies have switched their efforts into reducing the computation (cost of a producing a result) as much as possible without being too obvious.
What was an exponential slope in the quality of results over the last 36 months has now nearly flat lined.
Addendum: IMHO results have 'flat lined' not because the models aren't much more capable than a year ago, but because conserving the enormous processing cost (of an over subscribed user base) supersedes the goal of following the user's explicit instructions (e.g. especially if that means more processing cost) to generate the best results.
Whenever I give feedback on something, the answer is just “let me tell Claude”. The person has no understanding of how everything works, and most of the code reflects that.
The other day he hardcoded in a demo mode, simply because he didn’t even know how to set up a local environment and set environment variables. I’m confused as to why Claude didn’t even knew this, but it might just be the prompting.
I limit LLM usage myself, and if I do use it, I try to use it on extremely specific tasks. It’s the only way it works for me.
I honestly don’t understand how all these companies are getting away with generating AI code. Even in a small project I quickly fall behind on my understanding of the project.
Now these people can thrive because LLM coding encourages the incurious and punishes the deep thinker.
Now your feedback is just another prompt for them, the code might be slightly better, but the person learned nothing from it.
On the other hand my buddy is spending $10k in tokens a day on agents to build something. He's a very smart guy, former developer so it's not just AI psychosis talking.
Still trying to figure it out. Not that I have $10k to spend.
Just because we work with computers doesn't mean we don't take, er, social-damage. Or perhaps parasocial damage, in this case.
I got into programming because the problems of programming were interesting to me. But if the problems go from "figure out why this calculator is off by one in France" to "Get this LLM to stop spamming cutsey emojis", then maybe it's time for a career change.
Like when I'm trying to get it to create an image, and the first pass is beautiful, but ten different request to modify it, with different phrasing and even example images, produce the same image ten times. Or when you tell it not to use a cheap hack in AGENTS.md about six different ways and in your prompt, and it still does it again, and again.
It's like arguing with an idiot. And THAT gives me burnout.
Also: I've never once seen an emoji in LLM output. What are people talking about?
On the “pre” side, the specification of the problem becomes much more important. On the “post” side: QA and verification that the change has its desired effect, and no ill effects, also becomes much more important.
Sure, these are the next things to be automated, and people will try, but it’s easier on the backend (testing/verification can be automated) than the front end (the spec will be human-written as long as someone cares = forever for brands that matter), there will always be a need for humans on the specification side.
Even projects that used to be challenging enough to impress people with your skills can now be built in 10 minutes with AI just by describing what you want. It's an incredible shift, but it also changes how I think about the craft and what it means to be a good engineer.
The productivity drive and the sheer feature set you can generate in record time makes it easy to forget proper sdlc hygiene.
… but I am almost certain I’d never have developed those in the first place if I hadn’t spent 25ish years programming on a bunch of different platforms and setting up servers and networks and all that, without LLMs.
I dunno how you make another “me”, now, while before lots and lots of programmers naturally ended up as someone with skills and knowledge like mine, and those skills seem super useful when writing code with LLMs.
As a long-time engineering manager, PM and, eventually, product owner my response is, "Congrats! You've just been promoted to management." :-)
As a new manager, your first challenge will be successfully delivering commercial results using only a team of 'differently abled' new grad interns. Don't complain, new managers don't get to pick their first team! To be honest, these guys are more like alien brains raised in a vat with no direct senses. They've only ever experienced a data feed of the internet and, oh yeah, they get near-total amnesia a few times a day (but maybe you can teach them to write notes for themselves). They also have ADHD and are somewhere on the spectrum. But don't worry because what they lack in common sense, experience and intuition is offset by having a sort-of photographic memory and a willingness to grind on a problem 24/7. You should be fine. Good luck, we're all counting you...
Including myself as well. It's how you grow into the role.
“Look at the first letter of the prompt, and use the style of the matching literary stylist enumerated here:
For English:
A — W. H. Auden
B — Bill Bryson
C — Italo Calvino
D — Joan Didion
E — T. S. Eliot
F — William Faulkner
G — Gabriel García Márquez
…”
I certainly don’t see emojis any more.
My mind still can't function well without having knowledge about everything.
Anyone else working on something like this or know of any projects attempting it?
I've taken a bit longer than I wanted but it will be open sourced soon.
It's a durable orchestration engine that takes in specs/requirements and coordinates agents externally (meaning the engine drives the loop, not an agent) until the work is fully implemented/verified and reviewed.
It's meant to be used with any harness as basically the last step. You plan your work with whatever LLM you use and then hand off implementation to the engine (through an MCP server or other surfaces)
It can use your OpenAI/Anthropic subscriptions or any other provider and you can mix and match models across implementation and review in any way you want with fan out for parallel reviewers and more.
The goal is to produce high quality unsupervised code that matches your requirements and is reviewed throughout the implementation rather than at the end only, so that mistakes don't compound.
https://engine.build if you want to get notified when it releases.
Directionally if what you're doing is straightforward it's an amazing experience to be able to slap in an epic planning document and wake up the next day to it being "done", with a big asterisk that done-ness is directly proportional to how good of a spec and how good of a model you were using.
That being said, these days if you use Fable, slap in an epic planning document, and ask it to run a workflow (be sure to specify that subagents should use, say, Sonnet, or wave goodbye to your wallet), it's almost as good as gastown/gascity but far more predictable.
Main blocker is I am using apps like Conductor and have lots of plates spinning at once. But that's on, me and I should try and start completing the last part myself.
The bots (all of them) seem to show patterns of overuse of specific phrases, words, and punctuation.
Some of those are the ones you mentioned. Another that I've been seeing lately is overuse of the term "gate", wherein: As a human, I know what a gate is. A gate is a thing that can be open, or that can be closed. It might be locked or unlocked. The path beyond the gate may be passable or impassable or nonexistent. The gate is just a gate, and the presence of the gate doesn't imply whether it is open or closed.
But in bot-speak, a gate only refers to a hard block -- an impassable construct. Like a fence or a wall, or even a lava-filled moat.
But while a lava-filled moat is intended to be impassable, the bot uses "gate" -- a thing that is designed to be passed -- to describe that same kind of obstacle.
That's misuse of the term, I think, based on decades of dealing with gates in reality: Usually when I encounter a gate that is closed, I just open it and walk through.
I do have instructions that tell the bot to avoid that usage of the word and it ignores them sometimes anyway.
But "gate" is just today's problem-word that comes to mind as I write this. Yesterday, it was something different. Tomorrow, it will be something else entirely.
The overall pattern here is that of gratingly-repetitive bullshit-grade jargon that doesn't fit to begin with.
"And that's the real, no-nonsense truth!"
I've often had to paste its output back in to ask it what it actually means. Weird.
I think the main thing is just fatigue. There's so little variety. Each model has its preferred idiolect which everyone becomes tired of due to ubiquity. That's the worst part. It's like always eating fast food.
It's impossible to undo some of these linguistic wobbles. Even if you could filter out 100% of LLM input, the humans themselves are learning to say "land" at a higher frequency now.
Its like when someone points something out a in picture you never saw and now you cannot "unsee" it ever again.
Before, they could stay in thinking mode for more than 7 minutes. For example, "find a source for this claim" would search, analyze, and self-adjust the query. Nowadays, even if I push for it, I cannot make these tools work for more than 30 seconds before they give generic answers, even in "Pro" mode.
How empirical are your comparisons of new and old outputs?
Hell, the Opus 4.5 moment was only last November, and that was when agentic coding and most coding CLI tools became truly first class options. That's a wild paradigm shift. Hell, GPT-5 wasn't even out (that's August of last year). Most people were using 4o. Their current offerings are wildly better for coding than 4o was.
But that's also let me use "agent" stuff longer, I guess? The better you were at knowing what you wanted and how to ask for it, the less of an inflection point that you got from Opus 4.5 or GPT 5.
Some of the highest-time-saved-for-max-ROI agentic problems I've solved to date were in September and October of last year with Claude or Cursor.
This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is the best way to keep AI on track and make those diffs you have to read clean and easy.
An even better way is to use a more strongly typed language and really lock it down, but you can use testing in any language. I feel like my background in TDD and "TATFT" has been secret sauce when working with AI
Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit tests need to be there, so the really really helpful tests never get generated. Not to mention the obvious issues with generated tests only testing is x = x, or needless redundant tests for the same thing, or them essentially testing basic features of the language.
https://github.com/dprkh/eventfs
It has good test coverage, mostly unit tests but also a number of end-to-end tests. I also made the LLM build a benchmark, which you can find at the bottom of the readme. It is obviously slow, but I thought that it is good enough to work. When I tried to write a 1 GiB file, I found that it broke down, and after writing half the file, the speed went to under one megabyte per second. Implementation is 10k+ LoC, and I have no idea what is going on there.
It is different though. Basically a lot of what I do has changed over the last 2 years. I totally get that a lot of people won't want to adapt though.
I save myself by skimming things like tests, templates, some UI. Anything cosmetic. But I have to read the majority of code that ends up on my back end systems.
In my personal experience, the ones most enthusiastic about LLM magic are those that can't code, but can now walk away with something functional if not quite the best code. Now that they can produce workable code, it will make everyone better. Yet, they have no idea how maintainable the slop is or if it's slop at all.
When you see a perfectly clear function or object that just isn't your style, you have to accept it and move on. Where there are concrete concerns, or it's unreadable, demand excellence, but treat it like a coworker, not an IDE.
My latest is, I'm really into fizzy/soda water and wanted my own continuous carbonator. My entire build from water source to tap with an ESP32 controlled pump, pressure, water level, cooling fans.
There were so many areas I made mistakes in my shopping cart and it found it - like Home Brewer likes 8mm lines but water filter systems like 9.5mm. Really optimized the versions from a simple on/off pump w/ float switch to effectively a full on PLC system. So many iterations gained by chatting with "someone more experienced". Once I get the parts I can build and have the software side running in less than an hour.
It doesn't make money, but man I really enjoy it.
Sorry, some of us have a joy for programming where the how is just as important, if not more so, than the what and the why. No matter how much people proclaim that the how doesn't matter to them, it isn't going to suddenly make it true for others.
The experience is much closer to working with an external API that you don't have control over and which simply doesn't do what the documentation says. Those have always been the most frustrating parts of programming, but at least previously you could reverse engineer the actual implementation to work around bugs. You can't even do that now because the "boundary" randomly change every day.
We’ve had 3 production incidents this week that slipped past CI because there’s a whole team that is just shoving out PRs without understanding what’s going out.
this AI bubble will pop. when it does you'll be hot stuff all over again.
Or people don't want to be reverse centaur keeping the clankers happily running. Instead of helping to solve users/consumers problem.
do you mean my enjoyment from building things? I'm genuinely confused by this response.
It's not surprising that if you have a hundred separate, isolated contexts working on the same business, that don't cross-talk and have no ability to subconsciously receive and collate, prioritize the thousands of signals we get from our work environment, that you end up shipping lots of incomplete or incompatible work.
Probably like eating. Having to eat less to loose weight isn't great. Eating anything you want without worries is great. Having to eat more than you want to gain weight, not great.
Peer Gynt Suite's "In the Hall of the Mountain King" made a prominent appearance, but so did Aqua's "Barbie Girl"
Just be careful about any legal implication of doing side-projects during work with work-resources.
is the critical caveat to “that’s not how I would have done it”. Basically, choose your battles because we all have limited bandwidth. So, it’s not really a perfect binary, but a taste that you personally develop.
The only time I look at code is when something isn’t right and I ask for a root cause analysis. The LLM will show me some offending code or code for reference or evidence and then I quite often say “well that’s dumb you should do it like this instead” but I never need to actually go into the files. I do sometimes look at a git status or git diff.
At least with agent-run tests I care about loop speed a lot, but I care about complete coverage more, so having the odd heavy weight full stack integration test is fine, I think.
I actually have a public (AGPL) example here: https://github.com/pgdogdev/pgdog/tree/main/integration/sql - pgdog is particularly testable since it is trying for complete transparency, so you have a perfect oracle in hand via base postgresql, but it demonstrates the concept at least.
This is also part of why I like end to end tests that use actual UI flow, so I can watch it go by in slow mode before letting it loose fully automated.
I've been burned by this in my honeymoon period with unit testing (pretty much the reason it ended). These days, I prefer broader scope of testing, especially user-facing part. The users may be other developers or end users. I only do unit testing for tricky algorithms or math formulae.
They’re mostly a reflection of the current requirement of the project.
Why should I regret that? Why should I care about your purity tests?
This isn't a response I expect from people who are here for a productive discussion. I'm sorry that you are sick of hearing this, but I'm not responsible for making sure you only read what you find worthy of your own personal brand of respect. Instead of attacking someone for simply offering their point of view, in what appears to be a quasi-gatekeeping effort, maybe you should look inward and discover what is making you this upset toward a complete stranger.
__I cannot take away the joy you have for programming simply by stating what drives me.__
Look, I don't have a problem with your personal motivation. I just hate seeing it suggested that people should abandon their passion because someone else doesn't share it. There's absolutely nothing wrong with "I enjoy making something useful" just as there's nothing wrong with "I enjoy making something with my hands or figuring out how to make it". My problem is with "your enjoyment of that is invalid because I don't enjoy that, so learn to enjoy this".
0: Not really a statement of your drive, is it? More of a directive or suggestion.