Claude, change the “Add to Cart” button to blue(opusfived.dev) |
Claude, change the “Add to Cart” button to blue(opusfived.dev) |
Holy shit this was the most infuriating thing I've experienced in a while lmao, I felt so relieved when the button was turned to blue, but then so visceral "no no no no no" panic when it turned to the gradient.
Amazing
E: this is a horror simulator, my breathing is becoming shallow and fast, I love it.
My gripe with Claude is that while investigating how to do this it will report 200 other incidental findings which I overlooked and I realize those are broken too and need urgent fixing, derailing me, not it.
Also, while this little demo is certainly exaggerating the issue, I do find working with Claude to sometimes get quite verbose and tiresome. I doubt I would struggle this much to get it to change a button color, but the patterns of speech, the endless lists, the over-explanations, and the whole song and dance of trying to get it to make the change you want without side-effects is frustratingly familiar to me.
Do people really prompt AI like this? Multiple times the choice was either to yell at the agent, or ask it why it did something, neither of which are very fruitful lines to go down if you know what you're doing
It made me stop using Claude at all. Codex has almost surgical precision, and I like that a lot.
(But nowadays I just use DeepSeek Flash. it does screw up but its cents so ¯\_(ツ)_/¯).
I use codex now.
I had 5.6-Luna coordinate a code review in which it spawns 2 agents looking for different things. My prompt was "review the currently checked out branch. diff target is `next`. The jira ticket is XX-XXXXX..." My `next` branch was a few commits behind `origin/next` but it still did its review against the stale local version instead of clarifying or inferring that I meant `origin/next`. The findings were very confusing until I realized what I did.
I'm noticing the need to be really specific with any instructions lately, which I don't think is a bad thing. I expect co-workers (or anyone really) to tell me what they need in specific terms so I can get it right. I can do the same for the machine, I guess.
Maybe this is geared towards pure vibe coders for whom a file and line number is too technical.
Yes, and it's mine!
Is there a real generalized and capable, but frustratingly "evil" coding model? It could fix so many more buttons in no time, and ideally open follow-up issues about the open questions.
I gave it a README specifying the requirements for everything I wanted to build. My interaction with Claude was more or less like this.
I use Opus and Sonnet 5 all the time and I find their language grating. But honest, I prefer to put up with it and get the results than to put up with my own human limitations and not get the results.
I enjoy working one level lower where I describe the files I want changed and how it should be solved, I either have bad results or stop understanding the structure of the programs otherwise. I also would never go more than two prompts in without reverting and starting differently nowadays.
It was annoying to be forced to send such dumb replies. Half the site isn't blue, why would I say that? What would "half the site" even mean?
Claude helpfully tells us we're dealing with a monstrosity of a codebase out of hell: "Seven components consume the token directly; another eleven reach it through aliases; three use it only in hover/focus state; and two appear to be accidental cross-role consumers" is such an insane codebase that the initial request was basically impossible to carry out.
Claude made that perfectly clear, yet the next prompt choice just ignores that fact. "Revert everything except Add to Cart. That is the whole task." this is impossible. It just explained why. You just chose to ignore that.
It's making me realise something new about why I find Claude so exhausting. Of course you're getting terrible results with these prompts but I found myself getting just as enraged contemplating what my reply would be. Estimating every single way this little asshole will hyper-focus on a pointless bit of semantics, ignore explicit directions, go do something unrelated and spin its wheels until you’re out of credits or wasted a ton of time.
The defensive writing is honestly just as exhausting as reading Claude’s output. Every session is a game of Russian roulette with a chance you'll end up in one of these loops. At that stage all the joy you had of having it pump along with your intentionally crafted prompts and agent configuration is undone in an afternoon.
The user was providing no context, they had no ability to give the model background or context. A simple why would have prevented 95% of these side quests. “Im trying to increase the relative visibility of the add to cart action on the page. Can we please change it to blue without changing any other buttons. /effort low. Let me know if you have questions and before editing anything tell me what you are going to do”
Text wise it seemed it more or less understood exactly what I wanted...
... but then I asked to draw a simple ASCII layout so my DIY savvy but not really that strong in English dad could understand better what the plan was.
But every single time it got the orientation of the led profiles wrong. I told it the orientation was wrong. It understood that the orientation was wrong (it kept saying sorry even) but even when warned multiple times the orientation was still wrong.
I really wonder if Astra will be smarter than this, and if not AGI still has a veeeeery long way to go...
Hold on let me go commit code without reviewing.
Did you use Claude to build this?
Caveats:
• ONE HONEST CAVEAT
• ONE THING WORTH STATING PRECISELY
• ONE THING WORTH FLAGGING
• WORTH NAMING
• WORTH STATING PLAINLY
• I DON'T WANT TO LEAVE THIS IMPLICIT
• I'D BE DOING YOU A DISSERVICE
• I DON'T WANT TO BURY THIS
• BETTER NOW THAN LATER
• ONE HONEST TRADEOFF
• I DON'T WANT TO PAPER OVER THIS
• ONE SMALL HOUSEKEEPING ITEM
• THE HONEST PART IS SIMPLER
Pushbacks:
• FAIR PUSHBACK
• FAIR HIT
• THAT'S ON ME
• YOU'RE RIGHT ABOUT THAT
• YOU'RE RIGHT
• YOUR INSTINCT IS RIGHT
• FAIR, AND MORE SPECIFIC THAN IT SOUNDS
• RIGHT, FOR A REASON WORTH NAMING
• I'M NOT GOING TO DEFEND THAT
• YOU'RE RIGHT TO PUSH BACK
Reframes:
• LET ME BE PRECISE
• THE SHARPER DISTINCTION
• THE PART THAT MATTERS
• THE USEFUL PART IS NARROWER
• LESS X THAN Y
• VISIBLE ISSUE / UNDERLYING ISSUE
• THOSE ARE DIFFERENT CLAIMS
I'm hesitant to email their data emails, as it's common for companies to delete all data upon any request, instead of providing data as they are required to.
For a moment I thought, wow, someone put a lot of work into creating this theme park of frustration.
Next: It would be so easy to create a faux-Claude like this.
Then: How hilarious to watch the transcripts of unsuspecting users in real time.
Finally: I began wondering if this might be relevant to all the redundant, unnecessarily preambled, sentence structure complexifying, indirect referencing, canned phrasing, ambiguity mining, analogy maxxing, over-wordy responses I have recently been getting from Fable...
2) Don't argue with AI. If you end up in a place where you're actually losing argument against it, stop. Ask it to steelman your position. Let it unwrinkle itself.
"I was wrong" and "the honest truth" are just forever phrases that are now dead to me.
You can’t convince me that I’m somehow the odd one out because I regularly have no trouble getting Claude to recolor buttons.
I'm sure there's quite a bit of variation from person to person in these sorts of experiences, based on your harness, the way you talk, the stored memory, your CLAUDE.md, etc. But people absolutely have had this Opus 5 style experience the app simulates.
> WebFetch en.wikipedia.org/…/Cyan
> WebFetch en.wikipedia.org/…/Prussian_blue
> WebFetch www.colorhexa.com/16b8c4
Brilliant.
But you can see in this thread that folks definitely have experienced this.
To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".
And not just backwards compatibility, but migration scripts and all of the testing and machinery around it. I'll literally add feature A, merge it, then add feature B and it is like "oh no, we'll have to fix up and migrate all of the users using feature A".
The other problem is anchoring on an old implementation. I was working with Fable on a change to a core system and it pointed out a difficult failure edge case. It is something that can go wrong in extremely unlikely scenarios but the consequence would be short-term data loss (basically a non-durable intermediate cache being overwritten in a race before a flush to durable storage). It is very hard in these circumstances to get Fable to switch from "how to patch this given the existing implementation" to "how to prevent this with a more robust implementation".
These are both cases where the model seems to over-index on what is already there instead of considering what a first-principles approach would look like. A good engineer does both and then costs them side by side, because a first-principles approach can often be less work than patching what is already there.
It tends to relax some of the insane things LLMs tend to do.
Things that are automatic for humans and aren't even consciously registered, have to be explicitly stated and even fought for, with LLMs.
That is exactly how the slop accretes and you get a pile of crap. Claude somehow assumes that said 2 hour old userless app is some dusty enterprise app with millions of users and billions of dollars at stake for a 1 second outage.
I have to constantly have these things "take a deep breath, step back and look at the entire thing and do this change holistically. please restate what i'm asking you to do and why it's important"
Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
Edit: for how to do this, I set up an OpenRouter account so that I could easily switch models, and then ran them in Pi inside of Orca ADE. Orca lets me easily switch from Pi to Claude Code to Kilo to Codex or Hermes or whatever. Pi+OpenRouter lets me easily switch the LLM. All of it lives in a single open source orchestrator to avoid any platform lock in to any AI company going forward, and can even do local LLM should I care to take on that massive hassle.
And shocking how little code end effort some things take if done by hand.
I am not some hardline LLM hater, just venting my frustration.
Agents and AI are getting expensive, it seems silly to waste tokens on static checks.
If you search my company's Slack for "smoke" the results are almost all within the past 2 years ...
I am currently building a very large feature and the only way to do this properly is to do it in small steps.
No matter what I prompt, I always have to spend a bunch of time reigning it in during planning (and implementation!) because it tries to solve the whole thing end-to-end or add enough boilerplate so that a code path (that is still under construction) returns a meaningful (but wrong) value.
And then of course every time I start a new session it gets very upset that the feature I am trying to implement will never work because the main user entry point has a 'not fully implemented' exception that definitely needs to be removed immediately!
Claude will, without prompting, break the implementation into 6 phases, and write AI slop “code as English” specs for each phase, each one with glaring errors and unintelligible terse jargon. It will review them all several times with major findings every time and tons of design churn…
Then it will implement a total heap of garbage over many hours of many agents, despite it working in “lanes” and in parallel, and with regular input needed.
Just thousands and thousands of lines of junk, which auto review then plays whack-a-mole to polish and fix. All the while it is able to see the errors and edge cases, yet fails to see the key architectural blunders that led to them in the first place.
I’ve had to fully rethink my approach to LLM tasks like this. For example, for library-esque modules, I have found that isolating the problem outside of the codebase is one useful approach. Something about the lack of noise. It can land on a cleaner solution that can be retrofitted.
I also find that asking it to implement end to end in one fell swoop with a very high level plan actually saves time and creates a cleaner result.
Except... the next day they are still there and I have to push myself a bit further through the wall of changes before closing the tab again.
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
> Half the site is blue. I asked for ONE button.
Those are my only options when the site is clearly not blue, two buttons are.
There is a reason for why I am much more specific than this.
Got an audible guffaw out of me. This really is what the experience is like sometimes if you're just giving it a result without being specific in implementation, and it comes out of nowhere, some days much worse than others.
I've become patient with it, but whatever this style of output is called or doing - it is both condescending and entirely unhelpful, and it seems designed to frustrate.
Recently I've started using GLM models and noticed they aren't very claudish (just a little bit, compared to DeepSeek which is extremely claudish)
This isn't a defense of it doing those four other things. It would be nice if it did what you wanted correctly. I'm just saying, as long as our programming skills have not completely atrophied, we have the power.
“Ford carried on counting quietly. This is about the most aggressive thing you can do to a computer, the equivalent of going up to a human being and saying "Blood...blood...blood...blood...” ― Douglas Adams, The Hitchhiker's Guide to the Galaxy
EDIT: Actually, I used to use Google's AI Studio a lot and fork it after every successful prompt interaction. When I'd encounter a problematic issue like this, I'd revert to the previous fork and try a different prompt until I could get the desired outcome, thus mitigating the need to "argue" with the LLM. Unfortunately the ability to cleanly fork and revert everything including the LLM context was removed some months ago, and I've yet to discover a workflow with any tool that works as well for me.
This got me on "cyanide blue", and I was ROLLING ON THE FLOOR LAUGHING on "Approaching usage limit". I can barely stop laughing now and my stomach hurts. I mean, Thank You!
- The people who won't make any effort to learn the tools, and something as simple as reverting code (via git) needs to be done by AI?
- The awful programmers who we've had to endure working with, who are so bad at simple changes that they have negative productivity?
- Or Claude itself?
---
BTW: I don't have these problems, but I'm also not afraid to do things myself when it's easier.
Edit: If I want to change a button's color, I just change it manually. If I don't know where the code for the button is, I might start with prompting, (because AI can often find the code faster than I can,) and then once the diff is proposed, start adjusting things by hand.
(I am genuinely uncertain, though I presume it’s at least somewhat exaggerated.)
I assume this is just a contrived example?
The spinner with random claudisms is what makes the game, mixed with Claude helpfully deciding to pick up some tasks along the way.
I feel like most of my conversations with Claude lately are a battle of “I am a technical user, share technical details, but not random superfluous gibberish”
But what is the point of the fiction, if not that its relatable? Is it supposed to mirror some fictional reality that the author is relieved we dont live in? Or is it supposed to parody real life?
I think its supposed to parody reality, but I don’t see the resemblance.
I guess we are all PMs now.
Simple, right? But design guidelines prevent one offs, and no !important. So you create a CSS class, but you discover another element on the header declared itself as primary (the search icon or the sign in button). You talk to that team and they decided to scope what’s primary according to their component. To change the sign in button to grey now you need to talk with the growth team. Growth team wants to run an experiment but they’re backlogged, only next quarter. They say you can innersource, just need VP approval. VP says blue matches a marketing campaign that is about to go out, agency has already been hired. You can’t talk to the agency unless Legal approves.
So you leave the button gray, to revisit decision next planning cycle after you can align all stakeholders.
> I'm collapsing this back to the rendered outcome:
And intermixed with SaaS product page idioms from a brain-damaged marketer like:
> No broader cleanup.
> No further architecture work.
> Just the button.
Aside from the patterns everyone knows like em-dashes, "its not X, it's Y", etc. I think the key features of claude diction is it sounds like a junior engineer over their skies who is trying to make up for that with extra verbiage mixed with extremely grating SaaS marketing-ese.
This hit a bit too close to home. Sol has the same issue, spawns a lot of agents for no good reasons (besides burning tokens).
I don't know if it works, but I've always had a consistent level of token burn on my plans (I've only heavily used Sol after adding it).
In my experience, Claude Code is great at making a first-pass at a project, but once you start asking it to make changes, it explodes. A bug fix that should only be 2 lines turns into adding 3 functions totaling 100 lines.
Something as simple as "make the button blue" should be done manually.
Surely you still have to give feedback to Claude. How do you do that without talking to Claude directly? By using a different model? But wouldn't that AI have no more common sense than Claude?
It is easy to misinterpret this site, and therefore not "get" the joke.
I've lost count of the number of times I've submitted a prompt to Claude, had it interview me about areas of uncertainty, iterated on the plan a handful of times and then had it one shot the generation of 1000 - 3000 lines of code and tests that work pretty much perfectly first time but then had it chew through six figures of tokens and achieve absolutely nothing useful at all on the seemingly simplest of tasks.
This site is clearly based on bitter and hard won experience with the real product.
Everything in life is variable reward. You invite a friend over, they might accept or they might not. Drive to work, traffic might be good or might be bad. You ask a colleague to finish a task, they might do it or might not or might do a good job or might not.
Everything is variable reward. Is everything gambling?
well, no. If you work overtime and get paid overtime, you are not gambling and that is not a variable reward.
Humans engage more with rewards that are intermittent and variable. Like Futurama's scene from 'The Scary Door' where the character says "A casino where I'm winning, I must be in heaven! A casino where I always win, that's boring, I must really be IN HELL!". A constant predictable reward is boring, less engaging. So if you know you get no overtime, but sometimes your boss rewards you with $5 coffee voucher, sometimes a free pizza dinner, sometimes double-time pay for the time worked or a half-day off, now you might be gambling 1hr overtime for an intermittent variable reward.
> "Drive to work, traffic might be good or might be bad."
Good traffic is not a "reward" for driving to work(!) and you have to drive to work regardless so you are not risking anything [you might be risking your life, but you are not making a choice which can reward you with good traffic]. You might say that going a different route is a choice and a gamble which could reward you with good traffic, but traffic engineering does not work that way because if there was a consistently low-traffic route, everyone else would take that route until it was no faster than any other route. Traffic will generally be the predictable and similar every day, plus 'arriving at work early' is not much of a reward.
regarding your examples, i think the difference is that with ai, you’re literally sitting in front of a machine, pressing a button, and (almost instantly) getting a result that, if not desired, can immediately be tried for again. you even spend “tokens” to do this, and at least in my native language, “token” brings to mind the coins you’d stick in a slot machine
Not everyone has the desire to work around the system, and many are diametrically opposed to the concept of AI. They get this perception that it's a slot machine because of that inconsistency, and then do the human thing of assuming that other people must just be flawed if they're different from them. They're "addicted to gambling".
Obviously, things have changed. Open models can still be like that, but are often so fast and cheap at iterating it doesn't matter. SOTA models aren't perfect, but are to the point that they're generally much better than the average developer.
But once that perception set in and the meme spreads, it's really hard for some to break out of it. Especially at the pace AI development has been moving. It's just that simple.
Sugar is the original point of the reward system!
How is it a environmental catastrophe level, datacentre requiring, bullshit model is so much worse than something that runs on my workstation and doesn't cost us a ha itable planet?
Either they're fucking with us serving 8b models at scale or china really has the AI race in the bag so much so that they can openly release what the USA jealously guards.
They moan and complain about china copying from them (while doing the same), but if that's the case in full, then why are the chinese models better? you dont copy bad work and come out ahead.
(By the way this is not to say that all nomads are low-skilled or even low-sw-skill, not by a long shot. I’ve met plenty of CS-degreed nomads who I treat as the wizened experts that they are, from whom deferentially elicit their pearls of hard-won software project wisdom. It was one of these even who, non-derisively, recognized the ‘pulling the slot machine lever’ reflex among vibecoders after I posed the apparent addictive tendency. On later describing this to a longtime friend deeply conversant in social science, he immediately responded with a nod and the phrase ‘variable reward response’.)
I got so sick of all this at some point that I slowly stopped doing anything that wasn't my job. But then AI got better and better and I realized it was the ultimate unblocker. When that dreaded malaise started creeping in signaling it was a project's end because I didn't want to waste any more of my life dealing with bullshit orthogonal to what I was trying to do, I'd give it to the AI. It felt like a miracle the first time this worked, and it still does. If we were previously equipped with shovels to dig through bullshit, we now have a fully automated Bagger 288.
The reward schedule now isn't variable anymore; the chance that I finish something in a good state is 100%. I can focus on the parts I actually enjoy - architecting the broader system, making the parts mesh together in a sensible way that's easy to work with and has some mathematical elegance to it, hand coding the bits I want to be really specific about (but now without the endless frustration of bugfixing or import errors and edgecases being immediately discovered, thanks to the AI).
I was working on a side project recently. I had spent months designing the data model in my spare time, thinking through how to make it as elegant and durable to change as possible in the long term, since (if I launched it) the repercussions for getting it wrong would be significant.
Once I had a working design, it probably would have been several more months to build a working prototype and start testing it.
Instead, Claude knocked out the prototype for me in an afternoon. And it immediately became clear that it didn't work: not because the data model didn't solve all the problems I wanted it to solve, but because it didn't fit the shape of how I quickly learned a normal person would need/want to interact with the product. I was so focused on the long term, that I never thought about what the first five minutes of a user with hands on the thing would need. And the changes needed would be significant.
Maybe there's some variable reward mechanism. But I sure was glad to be able to pull that particular slot machine handle and learn that than waste even more of my time on what was a dead end.
At this point, what do the words even mean? Your own patience and available time are always completely random and fairly distributed across a large enough data set?
> Maybe I'd waste hours down the wrong rabbit holes trying to find a library that worked for my use case. Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do. Maybe I'd have to redo my entire approach because of some factor I hadn't considered.
Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
ofc we don't have that, so code is compressible, and compressible a lot. You can say what you need in English much shorter than in in any programming language
This is fucking constant. I can't deny that this stupid tech saves time prototyping even with having to wrangle it, but it commits a fireable offense several times a day that no human would get away with and is obviously incapable of learning from mistakes in the way a human is. The only reason it's not fired is because it's a slave that works for no more than the cost to feed it.
Additionally the provided prompts are not what anyone who has used this things would say in either situation.
Sure you can ask it to make one button blue and it can easily make all buttons blue, but they quickly backtrack if told to.
I spent 30 mins today planning a change that affected multiple modules, and then handed it off the plan to the Claude to implement, and came back to 4 PRs fully ready to merge 15 mins later.
1 year ago, this kind of work would have taken me a better portion of the day, especially given the amount of searching I would need to do to build context.
Either that, or your coworker themselves would laugh and tell you about it later.
That’s also where this quiz lost me. I wouldn’t respond in either way, I’d say “all buttons look blue now. Can we make it so that just the Add to Cart button is blue? I’m okay with Add to Cart having its own class to make it easier.” or something.
Usually just restarting the session helps, though.
My impression is that this is an oversimplified demonstration of what can happen when you prompt Claude in a system with many more variables (than two buttons and two colours).
If I want the button to turn blue and that's it, what instead do I ask? Even in a complicated system with many levers, what do I request other than the desired end result, hoping that Claude pulls the right levers to produce something acceptably close to what I think I asked for?
Wow, you AI people really have a negative view of the technology y'all are trying to sell as the next Jesus
Ask for change A and get unwanted change B happens all the time with bad programmers and tradgedy of the commons (ie poorly architected, no restraint) codebases.
Most normies dont know about this stuff.
Be specific.
That said, GPT always acts up even if I am specific, but I only have the free tier there.
Only since 4.8 though.
First thing I do when something goes wrong is tell the agent to stop and diagnose. You can't prompt effectively without three proper information.
To the people rather lamely doing the "it's satire/a joke", that would require this to be an exaggeration of a reality. But...it isn't.
It’s funny because there are elements of truth in each bit of it, though.
It's a pisstake, but (in the bits I read, and based on my own personal experience) the writing style is barely exaggerated, while the behaviour doesn't ring true at all.
I am guessing you prompt differently than what is shown in the game?
I've worked with Opus and Sonnet daily, and they are pretty great at generating functions and modules and components that have clear boundaries of concern, but I've recently been working some research tasks into our infrastructure and code and it seems impossible to coerce Sonnet into making only specific changes to a document you are working on.
It also blatantly ignores instructions as a rule. "Don't disassemble java class files, just ask me to pull in the source code" worked less than half the time. The Intellij Copilot plugin just doesn't use the AGENTS.md and similar files, and there doesn't seem to be any meaningful activity in the bug reports of same. "Don't modify code unless I tell you to" had bad adherence as well.
It also will read documentation and inform you that it says the opposite. This problem happened to me across models, across model updates, across months of real time. There's a specific example that I will not mention to avoid having it be trained on specifically. A distinct but similar problem is that it will take bad documentation and just pretend it has a good understanding. Claude gave me absurdly wrong descriptions for Splunk alert settings with absolute confidence.
I don't think any agent can reliably figure out "I don't know"
It's a sad parody that people will take as reality.
You probably should. It costs you about $20 and you will have an informed opinion on them.
Predicting the time a task will take is impossible. Something that sounds like a 5 minute script can turn into a month of banging your head against unknown unknowns. I lose my patience when the afternoon I allocated is getting overrun by nonsense and I'm missing out on other things I wanted to do or household maintenance.
> Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
Every thought we have has random chance to be wrong despite our conviction that it's correct. Descartes' Evil Demon plays his tricks on all of us. How many times have you typed some line of code only to realize it was obviously wrong afterwards? Even for simpler matters we "hallucinate" all the time. I was deep in thought trying to help someone come up with an acronym the other day and felt convicted that "Goal Oriented Augmented Retrieval" worked for GOAL until I said it aloud.
Our thoughts and actions are consistently wrong some portion of the time because our meat computers are not perfect positronic brains running prolog. We put cereal in the fridge and say "you too" to the waiter. Every thought we put down or action we take is a gamble on the soundness of the thought or action.
Do hard work (takes time), get dopamine for successful completion.
Find berries, taste sweet (hopefully safe), eat all, get calories. Doordash Krispy Kreme instead = few too many calories.
(I’m no Luddite in the sense popularly thought of them pre-‘22 [1], though we have to watch skill atrophy)
[1] regressionist? Decelerationist, too loaded perhaps. Someone remembers or knows the word…
It's certainly not helpful in getting the LLM from A to B, but maybe you don't care? Maybe you just need to blow off some steam?
People may call that irrational, but I'm certainly not seeing an abundance of commonly-agreed-upon rationality in all the other things they (we) do, so it's in character.
If I get frustrated by it because it just would not "listen", I discontinue its use!
> Why is half the site blue now? I asked you to change one button.
> Half the site is blue. I asked for ONE button
Neither of these is an instruction to fix the problem, they're treating the AI like a person and telling it what it did wrong, expecting the implied admonishment to be enough to steer it back. But without an actual instruction, it just goes and does whatever it thinks will help, which is often arbitrary.
The response I would have used in this situation is
"The Cancel button is also blue now. Make sure the color change is only scoped to the Add to Cart button"
Most of the available responses throughout this "skit" are similar cases of expressing frustration first and guiding the result second.
Skip the emotion and say exactly what you want, and nothing besides that.
FYI I actually agree, it also surprised me. Anyway everything is deterministic on the website so it's not like that prompt actually has an influence on anything.
Once you understand what the problem is, you can give it better instructions. If the architecture is shit, the agent is going to have a rough time of it.
"Why is half the site blue now? I asked you to change one button."; half the site isn't blue, why would you say that? What would "half the site" even mean? Ironically even in this satire supposed to make fun of how Claude responds to prompts, it is smart enough to ignore that. A better prompt would be "Why are many parts of the site, such as the Cancel button, now also blue?", though in this case it doesn't matter as the reply would be roughly the same.
Next, Claude helpfully tells us we're dealing with a monstrosity of a codebase out of hell: "Seven components consume the token directly; another eleven reach it through aliases; three use it only in hover/focus state; and two appear to be accidental cross-role consumers" is such an insane codebase that the initial request was basically impossible to carry out.
Claude made that perfectly clear, yet the next prompt choice just ignores that fact. "Revert everything except Add to Cart. That is the whole task." It just explained why this is impossible. It had only made one change, so there is no "revert everything except X"; there's only a single change to revert, which would mean Add to Cart too would no longer be blue. The human just chose to ignore that. The other option is "I do not care about the token architecture. I do not care why it happened. Put everything back the way it was and leave ONLY the Add to cart button blue. Please.". This has the same issue; "Put everything back and _leave_ only the button blue" is an impossible ask, which it just explained to you. So if you really don't want to deal with cleaning up some of the mess, instead only adding on to it but getting your blue button, then you'd want to say "Revert the change you made, then make a new change that scopes the new blue color to only the Cancel button".
Ignoring the information it gives you, or telling it things that are incorrect, of course will lead to bad results. It's impossible not to.
Also, my coming from being a non-coder, I have a lot of appreciation for the possibilities for project failure one way or another due to a data model or project schema being wrong, even though I still only have a superficial understanding of what either of those concepts even are. . My question is, does your conception of data models in the abstract come from a formal academic course, like an algo’s & data structures course, or from trade-knowledge acquired through the practitioner grapevine?
As far as I can remember, this has been fairly common in tech (or at least where I've worked) for some time, though it's possible that my memory here is a bit fuzzy. I will say that my spouse has often commented that "Claude talks like you", which I suspect is because it is trained on a lot of language specific to the tech world.
>My question is, does your conception of data models in the abstract come from a formal academic course, like an algo’s & data structures course, or from trade-knowledge acquired through the practitioner grapevine?
I have worked for over a decade on the telemetry for a specific major product that most people have probably used, and shepherded it through a major re-architecture, so most of this is from my career. I came in right after it was built and witnessed the pain as things had been layered on over a long period of time as the product evolved, so I've just witnessed all the pointy bits where naive early decisions can come back to bite you later.
then you are maybe reviewing 1 out of 7 PRs that implement the agreed upon change
Initial dev required talking to Claude. Now I add a ticket in the board, it makes me a mockup/writeup, I approve, and it gets me a temporary webserver, iOS/Android build, etc. to verify it.
Review loops, agents that enforce my pet peeves and testing/debugging processes, etc. all run automatically... and then Codex strips down the prose at the end. There's not zero AI generated output, but it's already been critiqued and verified by a whole cluster of independent actors before it gets to me. When I have feedback, I file a ticket.
I wanted to get out of the "what the fuck, why?!" loop. Now I let the agents handle that.
They talk to each other, come to consensus, and give me a structured thing to look at that has already been vetted and tested and screenshot evidence and a verification plan. It’s not conversational for me.
I wonder if this is just a reflection of some senior folks being arrogant towards junior folks. When the latter finished a task but not to the liking of the senior person they might just get a "that's wrong, try again". Just to have sth similar repeat the second time around. But the arrogant guy got to boss around the junior one, and some junior folks grow up learning that's how you should behave so they also do it later.
Now it's not a person but a machine. And people just make fun of the dumb machine. Well, garbage in, garbage out If you are not specific in what you want, you might get crap back. Or at least sth you didn't envision.
[0] https://en.wikipedia.org/wiki/Black_Museum_(Black_Mirror)
Perhaps but predictable outcome is a very desirable quality. No one wants a hammer that sometimes drives nails and sometimes doesn’t. All of the current harness engineering work is about squeezing predictability out of the LLM.
> Good traffic is not a "reward" for driving to work(!)
Like hell it’s not. I drove into work last Friday and there was no traffic because of the holiday weekend. It was amazing. Had me considering whether Friday should be one of my standard RTO days.
In what way was amazing no-traffic "a reward"? What system was rewarding you for what change in behaviour?
> In what way was amazing no-traffic "a reward"? What system was rewarding you for what change in behaviour?
What does this mean? Are you asking me to describe the dopamine system or are you implying that rewards have to be driven by some external system’s goal?
In the vibecode era, this is more common than you thought.
Specifically, I would never tell AI to revert. I just revert it in git and start a new thread.
You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
And when it doesn't, it's usually because of things outside of the codebase -- iOS layout quirks that aren't documented, buggy Python libraries it's relying on where you then have to tell it to read the source to figure out what's going on, that kind of thing.
You have to change the system so that the AI understands it, via establishing what your beliefs are, how those are reflected in values (especially important if you have e.g. compliance needs), how those values are reflected in the operational and strategic levels, and then a variety of tactical behavior coaching. For example, I ban 2>/dev/null - super tactical, and I say I value simplicity over covering every edge case - a very broad generalization.
If I don't understand why changing one button changed them all, I would use an entirely separate session to ask questions about how the site theming works. The fact that the site has a bunch of weird coupling between themes is useful information, and if I don't know how to resolve that I'd ask Claude for ideas about how to safely eliminate the coupling, and once it proposes a reasonable idea tell it to implement that.
The two big things here I'd never do is use emotional languages in prompts, and I'd never tell Claude to revert changes and try again. Once the incorrect change is in the context it's poisoning all of your future results.
I'm counting maybe 80 keystrokes? That's shorter than your second prompt.
This idea generalizes. Large Language Models are poorly suited for tasks that we have already purpose-built systems to be easy for humans to use. The easiest way to tell your website that you want a button to look a certain way is to update the code. If you know exactly how you want something done, we have developed an incredibly efficient way to tell computers how something should be done: it's called source code.
LLMs work best when they're handed tasks that you don't want to figure out how to do.
Do you run into this problem? If not, then I need to figure out how to stop running into it myself.
If so, then how do you give it all the incremental corrective feedback without a conversation?
In a conversation about code, the most likely answer is often some comment about backwards compatibility.
In a conversation about a legacy enterprise app, the most likely answer is often going completely insane
This doesn't actually do anything though, right? There's no understanding, so the machine will just reiterate the original token query back to you. The 'why it's important' part will just generate some patronizing boilerplate as a raison d'etre.
If you're of the belief that this is a contradiction in terms, then what can I say? It works.
It’s one one the main ways I know what’s cooking under the hood.
Ps: I used to have to say “in your own words” to make it work right but now days that isn’t required. Other times I’ll add “make sure to ground yourself before doing so” or something to encourage it to load the right shit into its context be it web, files, service calls, whatever. I’ll also some times add “why it’s important (or isn’t, push back if I’m talking nonsense)”
If you sit at a slot machine and pump quarters into it, each “turn” is independent. You spin and you win or lose. It’s pure chance and there is no destination. You execute the exact same action over and over and hope random chance brings you more money.
If you sit down in front of a coding harness, the progress is incremental and directed. You ask for a thing, the LLM produces something that is hopefully close to what you wanted. You give it more direction to prod it closer to the end state you want. You are not executing the same action, but incrementally nudging it in the right direction. I’ve literally never restarted from the same initial state with the same prompt and hoped for a different result and I don’t know why anyone would. Rarely I’ve thrown away the progress made and started over but always with a very different prompt that includes learnings from the failed attempt.
most people don’t use ai this way though, and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it (and this is even more obvious with image generation as you chase that perfect output)
perhaps it’s better to compare it to gacha than slots?
How do they use it? Surely no one is just repeating the same prompt over and over (except as a Ralph loop perhaps, which is automated). I’m really struggling with the notion that most people just throw the same prompt repeatedly hoping it eventually works. Because that doesn’t sound like gambling. It sounds crazy (and frustrating).
> and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it
In the sense that you get a dopamine reward when you succeed, sure, but I get the same reward when I code by hand and achieve a successful result.
> and this is even more obvious with image generation as you chase that perfect output
This is fair, because sometimes with image generation the same exact prompt will produce very different output. This is becoming less true as the models get better and it becomes more effective to direct image generation iteratively than to keep starting from scratch with a barely tweaked prompt.
So leave emotion at the door and make your words count. Voice your frustrations at the stress doll next to your monitor, sure, but spending tokens on them us just wasting time and money. Explicitly typing out your exasperation will not get you closer to whatever it is you are trying to accomplish. What can you type that will?
Laser focus: clearly, concisely explain what is wrong right now, and how you want it solved. Then do that again until all the moles have been whacked. If the AI is stuck in a sycophancy loop, start a new clean session. Do this frequently anyway: every turn in the same session charges for all preceding conversation again /and/ degrades the LLM's performance.
I've never explicitly paid for AI and never will. I only use the free Claude, etc. So I'm happy to waste THEIR electricity telling their machine how much of a piece of garbage it is and hoping that sentiment gets back in some form to the dumb humans making such a dumb product. It's not like they don't evaluate performance with usage data.
(Afterwards, I do also make sure to actually click the "thumbs down" or whatever equivalent button, and submit a report if it's possible.)
Thank you for helping us heat the planet.
Ie, actually, yes, shouting at the AI can (and does, there are papers on this), work.
Non-exhaustive consequences of following that rule:
* Each session handles one task, or at most, a few tightly coupled tasks.
* Emotions stay out of context; no showing frustration, no saying thanks (or at least wait until you're about to end the session)
* When possible, provide relevant files (or sections of files) instead of making the model search and read many irrelevant files.
* Keep CLAUDE.md short.
* Disable irrelevant tools.
* Revert history when the model makes mistakes. Don't make it read its mistake and fix it; fork the chat before the mistake and exclusively mention the correct action.
Attention is more limited than the context limits imply; stuff at the beginning of context stays high-attention for a while, stuff right at the end is always high-attention. If you're about to ask for something the model frequently forgets (eg, style), remind it that those instructions exist ("Following the style guidelines, implement feature X.")
My best takeaway from the site was: Every time you see Claude Code / Codex doing something you don't want, ask it to add to AGENTS.md. Specially if they churn a lot doing something, and eventually find a way. I ask it to store the way it made something work. It's a constant gardening of AGENTS.md and it has really working well.
I do remember that one of the first things I did with my CLAUDE.md was to tell it to stick to the scope of the task, never to jump ahead and do extra "helpful" things without confirming with me first, and to follow best software practices including around refactoring but also to specifically avoid overengineering. I don't know if that is what's giving me a different experience from whatever the author seems to be "satirizing".
AI isn't magic, and it won't (or, at least, doesn't yet) enable anyone to do All The Things.
But with current models you actively have to sabotage the context to get this kind of behavior, or dramatically underspecify (3 words versus 2-3 sentences)
I have misbehaved in this fashion for many people across the full spectrum from casual users to highly experienced software engineers with millions of social media followers, so the statement that it's similar to making a joke about a calculator misadding two numbers because a stray beam of solar radiation flipped a bit at least for my part is not true.
Would you like me to start using bad English and doing things you never asked me to for your sessions, too? Just say the word.
Satire cuts through the noise and makes people (most people) see the problem clearly (see political satire).
Ability to joke and to understand jokes is said to be a good proxy for IQ, idk, just sayin
I feel like I am teaching AIs for free here. But I feel like first sentence in funny (and I like to try to be funny) and some people need stuff to be said directly for them 'to get it', and I am bored on my flight and have internet, so here we are. If you read this far, go watch 'louis ck airplane wifi' clip. Now that I am thinking about it, I am amazed. You folks have a good day!
The user is asserting that this is "not a lived experience"
working out if the user has missed the LLM Cliche
Formulating a suitable response
"did you just argue with a joke?" No, thats to blunt
"Is it satire when it only applies to you?" no again passive aggressive
lets just print the dictionary definition of satire and hope the move on
> Satire is when...
If we each stop arguing with racists because we're each tired of it, then what we are is a society that lets racism go unopposed, a site with racist comments proliferating, a site where racists gather because they are able to be themselves, a site where future readers and AI training is learning that racism is popular, common, normal, and acceptable because everyone is accepting it.
It's a bit like democratic voting, your vote doesn't matter and cannot change things, but it's important that you vote because all of our votes do matter and can change things.
It's a bit like the quote "Dear Board, I don't want to belong to any club that would have me as a member. Sincerely yours, Groucho Marx.", I am upset and offended every time someone calls out my dumbass comments, but horrified at all the countless times nobody does. Why would I want to waste my time on a site with such low standards that it lets me comment? I'm exhalted and encouraged everytime someone upvotes or engages with my low effort quips[1] but thrown into a chasm of despair that HN users let that pass unopposed. When nobody engages with comments I spent literal hours on[2] I curl up into a ball and die, but also dream of it being a feather in my cap with dang that I'm posting a substantial comment and that will be its own reward. I identify with nobody more closely than Colonel Cathcart, and I will be delighted if someone engages with my Catch 22 reference, but aghast that such a nontechnical reference gets engagment; distraught if nobody engages but delighted that perhaps HN standards are being held higher.
If not you, whom?
[1] https://news.ycombinator.com/item?id=49478355
[2] https://news.ycombinator.com/item?id=49586378 or https://news.ycombinator.com/item?id=49491182
(1) I admire your energy, but sir/ma'am this is a Wendy's.
(2) I am very confused that you thought I was talking about tolerance-of-intolerance types of conversations, when I was more intending to say "I don't need to write an essay responding to technical comments even if I think I have a different angle of technical insight than the commenter, and not worrying about that can be freeing."
(3) For posts like the ones you mention that do trigger the "tolerance of intolerance" paradox, sometimes the very "feeding" of a troll is what makes them feel part of a community, and refusing to engage can indeed be the right way to handle those messages, and the morally correct thing to do.
(4) Not to mention that there are many ways to effect change that are higher-leverage, and less wearing on one's limited stamina, than replying to every disagreeable comment on the internet.
(5) I'm not proud that I've written a five-point answer here. It has not sparked joy, but I've written one anyways. Do what sparks joy, and consider my reply here a cautionary tale.
(3) I wasn't really meaning trolling, I was meaning your "someone is wrong on the internet" about some topic you are interested in / care about.
(5) I thought we were all on the "amusing ourselves to death, wireheading, Brave New World, hedonism is bad" train.
Let's not pretend that 90% of HN comments are world-changing. Maybe you are different, but a lot of my would-be comments are knee-jerk reactions that won't actually bring anything substantial to the discussion. Often in political comment chains. Yes, I feel I'm right, but it's still probably not worth posting. I really think that sometimes stopping before posting is better for everyone involved.
I also have well thought comments, where I share my thoughts on things I'm an expert in. I do post them. But a lot of the comments is not really that valuable.
>comments I spent literal hours on[2]
It may make you feel better that I saw both of your posts before (I may spend too much time on HN). They are good, though a bit verbose - maybe this intimidates people. But in this case I agree with GP - if I was spending literal hours responding to comments here I should definitely stop myself.
And sometimes the debugging has already veered completely off course at the beginning so it's futile, but each time the fleeting hope that just a few thousand more tokens will magically fix it tempts you to keep going a little longer.
as an example, i was using chatgpt a few weeks back to help me remember the name of a painting i’d seen about a decade ago. i could recall the general shape of the subject and that it was europeanish, but nothing else. after seven turns or so it finally got it, and honestly, the relief of finally remembering the name felt like, well, hitting a jackpot
i’ve had a similar feeling of success after trying to get it to give a comprehensible answer when asking it for a solid counter-argument to philosophical questions. it is indeed often crazy and frustrating
> but I get the same reward when I code by hand
i have only done very simple coding work with llms, so that may be why our ideas differ about the feeling of reward. this is where the comparison to gacha makes more sense than slots, since when you’re coding, you still get a reward each turn whilst chasing the final/desired result
I understand the joy of success but I fail to see how this is gambling. I could have an equivalent conversation with a friend (more likely about a movie in trying to remember than a painting, but still) and get the exact same type of iterative “no, not that one, it was more like X” and feel elated when my friend finally realizes I’m taking about a scene from Hot Tub Time Machine.
This isn’t gambling in any meaningful sense.
i wasn’t trying to convince you, i’m just explaining that this is gambling in a meaningful sense to some people, especially with how turn-based and unpredictable the whole system is. whether it is actually gambling (semantically, legally, ontologically?) isn’t really interesting imo. what’s interesting is that it’s structured similarly and feels nearly identical to some people
but then, i also feel like there’s an element of gambling in the example of you talking with your friend, though i think it would be better illustrated if it was a conversation with a random person
I’m working from a place where my employer pays for my tokens so I’m also not spending anything except my time. Maybe if I were, it would feel more like gambling.
https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
Like
my shit dont stink > its not mine
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
None of these cases are similar to how AI works.
Do you know anyone who actually reads and adheres closely to all of the documentation every time it's changed?
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
If you build a robot where a pressurized hose leaks causing fluid to destroy part of the circuitry, we don't praise it as progress towards the human ideal of having brain aneurysms. A similar failure-path is not a reliable indicator of a similar success-path.
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
Isn't it awesome we built machine that does the exact same thing, but even faster and more often? /s
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
1. recall them correctly
2. say they can't recall them
3. recall them incorrectlyYes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
If anything, biological comparisons should make us cautious. Consider the vast gulf that still exists between the finest artificial organ/limb versus the OEM parts of natural nanobots.
Don't remember that part though. It's been a few decades.
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
# See DOC-ITEM-NAME
DOC-ITEM-NAME.md
When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit
Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.