I probably got the whole thing done in 5 prompts and still had enough brain space to vaguely follow along the presentation. Before this kind of thing would have taken 20-30 min of heads down coding. This would have been a strictly "after work" project which means I probably wouldn't have done it (my real side projects and family need that time more than this analysis did.) That's the kind of thing that an experienced programmer can get out of vibes coding.
I could have written that code. I'm sure there is a program that can do exactly that for me that I could have downloaded. But one prompt, a few tests to make sure it wasnt going to nuke all my files, and within 5 minutes I was completely done with the file management. For this type of stuff it just saves time.
I also get better results when using a prompt window built into and ide than something detached like the openai website.
But then again, I have been doing this for 10 years; that is my edge. Same exact stack (Django + boring frontend). I know the ins and out of my stack, quite obviously every single day, I see AI go into a direction that I know is going to produce a huge footgun along the way. I can just see that up ahead, suggest a different approach, and continue. IF I was entirely new to this, I would end up building stuff that breaks down after weeks or months or investments, not knowing when things went wrong, or how to go forward. Regardless, I feel like my time has come, and I am definitely spending 95% of my time just prompting the AI versus writing actual code. Even for the most minor changes, like changing a CharField to a TextField, I don't even want to open the models.py myself. In Cursor, I am averaging 5000-7000 fast requests per month, because in terms of ROI it pays off. I am looking forward to this getting better.
You'll see smaller team sizes at first, then continuing to shrink as individual positions get a higher workload and spread of knowledge.
I think "Vibe coding" is probably a canary for all of this so it's worth paying attention to what a non-programmer can actually accomplish. This creates narratives that get picked up by managers and decision-makers.
The capabilities taking user input will surely be hacked eventually, so I think those are a non-starter, not-for-nothing because of bitter, laid off developers wanting to see you fail.
In other words: "programmers on whose work the LLMs were trained will lose their jobs"
Alternatively if a “non coder” creates a project by vibe coding and it fails, maybe that failure happened faster with lower costs (especially their time, if it’s their own project) than if they’d had to go get financing, hire an offshore dev or two, and go back and forth for a few weeks or months.
Vibes are high on vibe coding.
Friday AI report: A user was observed seeming to struggle a bit with X; we had a pain point discussion; they suggested some documentation and UI changes; they were confused, but further discussion turned up plausible improvements; we iterated on drafts and prototypes; I did an expanding alpha with interviews, and beta with sampled surveys, and integrated some feedback; evaluation was above threshold with no blockers; I've pushed the change to prod, and fed the nice users cookies.
Idk why terrified, vibe coding is nice but everyone who developed something bigger than a toy knows that code is 5% of the task and never was a bottleneck. Its not like faang employees write code all day long, or even half day.
Ah and you need to make sure it doesn’t nuke your db or send weird email to your users because someone prompt-engineered it badly.
I'm terrified some colleague is going to vibe-code a product and use it to get promoted, then move to another team and dump the project on me to maintain.
In that particular situation you will have easier time switching a job(that usually pays more than promotion) than your colleague.
I feel people/hypers keep rumbling about many things and way too often seem they have no real life experience.
The biggest problem I’ve seen over 20 years long career is people and the games they play (on different levels of leadership and management within an org) and their inability to agree to and then verbalise what on Earth they want. No matter is it a greenfield or some digital transformation project, they a plagued by fear and self-interest (of various kind).
And even if/when they identify a problem (e.g. games players) it becomes a risky cutting the cord since the one needs to identify who to rely on to clean the mess and their confidence caves in (often) while resolving to “let’s layoff x number of ‘leaf’ employees” is safe scared-with-no-vision leader move and it looks good on the stock market.
Software engineers (generally) kept delivering despite all that for years across the board (generally). And, generally, kept their values and principles and that bothers this managerial class a lot it seems and they (in a way) can’t wait to stop paying “those nerds” big salaries and that’s why those often low values and unprincipled people can’t wait to see our backs and are getting hyped about this “AI will replace SD/SE” mantra.
All this above is obviously generally speaking but yeah we people tend way to often to misplace our focus and solve lesser priority problems, and this “ai replacing engineers” is one of them to a good degree.
And for the majority of software developers there is nothing to worry about. The sheer keep-knowledge-up-to-date demand this industry put on us primed us to be by far the most able professional group to jump into a career change in no time.
Which other group in huge numbers can just sit down and learn and work for 10 hours a day 7 days a week and not complain or get emotionally disturbed (too much) and get the thing but us.
So in the worst case scenario we will be fine, others should be scared if we in numbers pick their industry to move into.
I still find myself building more building block style. „Make me a python function that does X“ and then stringing those together by hand
1. Context sizes are going to grow. Gemini with 2M tokens is already doing amazing feats
2. We all agree that we should break bigger problems into smaller ones. So if you can isolate the problem into something that fits in a LLM context, no matter how large the larger software system is, you can make a lot of quick progress by leveraging LLMs for that isolated piece of software.
Engineering - working to constraints, including user needs (ie Product Management) is forever.
You want to have an LLM help you crap out a script, sure, but you mean to tell me you'd seriously consider using an LLM for a production systems that affects real people that deals with real people's real data and call yourself a software "engineer"?
Engineering is about designing systems that serve society and provably meet well specified constraints. You don't want the god damn bridge to collapse under load. If you feel comfortable using an LLM to "engineer" a software system, you ought to feel comfortable letting civil engineers "vibe out" their bridge designs. God this hype cycle has just made a complete mockery of this whole industry and I have no respect for the clowns pushing this shit.
The bridge collapsed? Oops, software error, can't do anything about it!
* Slack is down
* StackOverflow is down
* their default docs website is down
* their IDE of choice is being buggy and so they have to use
something without autocomplete
> A radical approach to the complexity problem has been to suggest that the easiest way out is simply to make the machine do everything; i.e. automatic programming. [...] this approach does seem seductive, it is our estimate that it will not in the short run produce results of much value to the designer of [...] large scale programs
> the belief that man-machine interaction can be a symbiotic relationship in which the overall productivity is greater than the sum of the parts.
> how a knowledgeable computer could help an already competent programmer. It has been our experience that we can produce better and cleaner code faster when working with a partner who shares our understanding of the intentions and goal structure of our program. We, therefore, believe that the appropriate metaphor for our work is that of creating a program with the capabilities of a junior colleague working on a joint project. The program should know the problem domain, implementation techniques, and the programming language being used fairly well. It need not know everything in advance; it can always ask its senior partner for advice or further information. Furthermore, this program might well be capable of paying more attention to details, of writing trivial parts of the code, of checking that certain constraints are satisfied, and even (in some cases) of cleaning up a large system after it has been put together.
> First Scenario: Initial Design > I'd like to build a hash table `O.K. youll need an insert, a lookup, an array, a hasher, and optionally a delete routine.` The P.A. knows the main parts of a hashing system.
> parallels [...] between understanding a program [...and...] natural language. In both cases, a key component in the understanding system is the background knowledge base, which establishes a context for understanding the semantics of the particular utterance in question. The huge problem in natural language understanding research is that if you try to advance beyond conversations in toy domains like the blocks world, this background knowledge quickly amounts to having a common-sense model of the whole world of human existence. Unfortunately, building such a representation of the world is exactly the central unsolved research project of the entire A.I. community.
> The transition from tab equipment systems to the modern day computer utility, exemplified by MULTICS, has taken little more than two decades.
Understanding LISP Programs: Towards a Programmer's Apprentice (1974) https://dspace.mit.edu/handle/1721.1/41117
It completely misses nuance. Are any of these apps actually useful?
I'm not sure how this is any better than jamming a bunch of Wordpress plugins together to kinda get the software to do what you want.
In developed counties developers are quite down to Earth decent earners and generally are not playing “too cool for school” games.
It’s slightly misinformed comment or, better to say, context dependant to be facts based.
if you know how you want something done, tough luck. LLMs, even the "really smart" ones, still often do it "their way". they use "their style" (whatever the most common way to write something might be) and "their preferred packages" (what ever the most common ones for the language are). i remember someone told me "hey dude try vercel's v0 it's so good" and i asked it for some basic svelte code. it spat out react.
if you are modifying an existing, non-AI codebase, it's really annoying for the same reason. if you have a preference for specific design patterns or code style, it's unlikely to work well without substantial prompting and re-trying.
they still can't really fix bugs. syntax errors sure, but actual time-costing logic bugs? figuring out lifetimes with rust? forget about it. all they do is add freaking print statements and say "try these things to fix it." no. you're the robot, you work for me, you do it.
they suck at functional languages/haskell. like they're really just bad.
lastly, they're interns, not employees. interns require hand-holding, supervision, and verbal abuse to get anything done right. bots are, for now, the same. they impose a cognitive load when you want something of any importance done: you can't actually trust anything it outputs, at all. you have to go re-check everything it does.
i remember a few days ago i wanted to parse a bunch of UDP packets from 10-20GB daily pcap dumps. I gave it the spec for the message format as a PDF and said "write this in rust", along with the existing (functional but slow) python implementation. this should be a simple case to apply an LLM: simple, routine, boilerplate code that can be next-token-predicted fairly simply, but still takes an annoying amount of time to type out. unfortunately it screwed up multiple times. it failed to use the pcap parsing crate (even when i supplied docs) because it probably wasn't frequent in its training corpus. more importantly, it just miswrote constants. like it would get the constant for length-checking a certain message type wrong despite it being plainly specified in the spec and the python version.
LLMs are cool research tech and I have friends who have used them to learn to write Python scripts and react webshit. in my opinion, they are of little value for "serious" programming. i realize that's an annoying and vaguely-conceited term but it's the best one I can think of at the moment. i look forward to when they actually work well.
in my opinion, a good improvement would be focusing on writing in at least somewhat-verifiable languages or writing s.t. pieces are verifiable. robot translates your request into rules, robot 2 writes the code from rules, SAT solver checks the check-able chunks for validity while robot 3 is specialized in checking unverifiable "connection points", use of side effects, etc. the "intern" problem is by far the biggest of what I've listed and this is probably the best way to solve it. once that's done, we can hopefully let these chug for a while until they get it right rather than giving users crappy output.
oh, and they MUST be tuned to be capable of saying, "I don't know."
https://www.youtube.com/watch?v=_pLlet9Jrzc&list=PLrEMgOSrS_...
And here, Redis bugfixing with Claude Sonnet:
https://www.youtube.com/watch?v=rCIZflYEpEk&list=PLrEMgOSrS_...
Some questions I had successfully answered recently:
> "I would like to animate changing a snippet of code. I'll probably be using Remotion. Is there a JavaScript library that can animate changing one block of text into another?"
> "In Golang, how can I unit test a http mux? How can I test the routes I've registered without making real http calls?"
> "Hello ChatGPT. I have an application using OpenTelemetry. I want to test locally whether it can collect logs and metrics. The application is running in Tilt. How can my local integration test read the logs and metrics from the OTel collector?"
That's:
sql query, slack webhook api docs reading, ecto query for oban jobs with complex para filtering, oban job to run cron, cron syntax.
easily like a 2 hour job?
it took me 5 minutes with AI. then we decided to send the slack alert at 7am EST instead of 12pm PST. instead of doing all that math, I just ctrl+k and asked it to change it. 1 second change.
these gains are compounding. if you're an experienced engineer, you let go of minutae and FLY. i believe syntax memorization is dead.
Don't rely on the LLM for design. Start by defining some tests via comments or whatever your tools allow. The tab completion models can be helpful here. Start the interfaces / function defs and constrain them as much as possible. Let the LLM fill in the rest using TDD. The compiler / test loop along with the constraints can get you pretty far. You'll always need to do a review but I find this cuts out a lot of the wasted cycles you'll inevitably get if you "vibe" with it.
For Cursor specifically, you'll want to aim for prompts that will get you from A to B with the least amount of interactions. Cursor will do 25 tool calls (run the tests / compile, whatever) with one interaction. A little planning goes a long way to doing less and paying less.
A few examples from my experience:
- Here is a SQL query. Translate this back into our janky ORM
- Write tests that cover cases X, Y, Z
- Wtf is this code trying to do?
- I want to do X, write the boilerplate to get me started
- Reshape this code to follow the new conventions
on-topic: I have no desire to be a front-end dev but my friends don't know a front-end dev from a front-end loader, so whenever they need something thrown together i usually reach for Ghost or whatever. But now they want custom "web apps" - and that's fine, thank goodness for AI.
Most of my "LLM" use is remembering phrases, terms of art, and the like. I have used aider and Cursor, and they're about as useful to me as stackoverflow, i suppose. LLMs have definitely improved a lot, i don't get circular chats much except on Gemini, Gemini is the weakest LLM by far, which is ironic considering the extent of data they have.
Are you capturing this value? Are you getting paid 4x as much as before? If you are not capturing this increase in the value of your work, who is? If you're not getting paid 4x as much as before, which I doubt you are, why are you doing this?
> why are you doing this?
Because I love doing this? Both web and software development are passions, AI feels like a lever; and making my money with this is nothing short of a dream come true.
The main argument from senior folks is probably that vibe codes won't cut it for actual sizeable problem. There is complexity that can't be abstracted away just because we want to.
All i see are toy examples. great you built tic-tac-toe with a prompt. now what?
1. Build a computable model of some facet of reality to an achieve certain goals. 2. Realize a system that manifests the model, satisfying a set of other constraints, such as resource constraints and performance. 3. Ensure a community of system owners comprehend the key decisions made in the system and model design, and the degree to which certain constraints can be relaxed or increased and how the system can evolve such that it behavior remains predictable over time.
Basically none of that, in my view, is supported by LLM driven "vibe" coding. Sure your hobby project might be ok to treat like an art project, but, oh, I don't know, how about software for train communications, or aircraft guidance systems? Do you want to vibe code your way then? Do you want a community of engineers who only dimly understand the invariants satisfied in individual components and across the system as whole?
LLM fanatics are totally ignorant of the actual process of software development. Maybe one day we'll find a way to use them in a way that produces software that is predictable, well-specified, and safe, but we are not there yet.
That's hilarious! In 1974, that was decades from actually happening.
Have you tried the reasoning models? I find that o1 really is able to find logic bugs, even tricky ones!
my guess is for some applications they can, but even reasoning models (o3-mini-high, grok3, sonnet 3.7, o1, deepseek, etc.) often fail to fix logic bugs. note that this isn't necessarily a form validation logic bug I'm referring to but, say, a pretty in-the-weeds tool for cleaning and pre-processing data for ML purposes. my guess is basic business-logic-y type stuff is much more doable.
i haven't really found a good way around cases where it either just adds printfs or loops through the same, non-working fixes repeatedly. they keep getting better, they're just not yet below my epsilon for unreliability.
The most embarrassing part is everyone immediately going along with it. It is like an improv game or a man on the street interview where they ask a pedestrian about an event that didn't happen and the person being interviewed just acts like they were there and know all about it.
We had the same situation with TDD: can we give out succinct specs and ignore what happens in the code if the specs are met ? For anything beside Hello world, the answer was no, absolutely not.
It still mattered that the logic was somewhat reasonable and you were not building a Rube Goldberg machine giving the right answers to the given tests 95% of the time. Especially as the tests didn't cover all the valid input/output nor all the possible error cases in the problem space.
It's because there's a lot happening that we need to have simple blocks that we can trust and not black boxes that were already beyond our understanding.
- "I've never used this language / framework, this is what I'm trying to do, how would I do it?"
- the documentation for these libraries is ... not useful. How do I do X?
Followed by: "these parts didn't work due to these restrictions, tell me more".
(I'm currently using this one to navigate Unity and UdonSharp's APIs. It is far from perfect
but having *something* that half-works and moves me in the right direction of understanding
how everything connects together is much much faster than sitting there, confused, unable
to take a single step forward)
I'm reminded a bit of the days when I was a brand new Java programmer and I would regularly Google / copy-paste:
public class Foo {
public static void main(String[] args) {
}
}
if __name__ == '__main__':
run_me()
Though, I do agree that hallucinations have dropped dramatically.
That helps a lot whenever I am working off something I know will likely exist but just needs to be pulled from a company's documentation.
More toolcalls per interaction is typically a product of planning ahead, which in my experience produces better outcome. Each toolcall is a verification of the last step. Without those guardrails I find I waste a tremendous amount of time.
> I could have written that code.
Not knowing precisely your desired result, the benefit of going through the effort of writing a script is experienced gained and deepening one's understanding of the tools involved.
For example, assume this file structure exists:
.
├── 001name-eu-central1.ext
├── 001name-us-west.ext
├── 002name-eu-central1.ext
├── 003name-eu-central1.ext
├── keep
└── research
for file in [0-9]*
do
dest="$(echo $file | sed -E -e 's/^[0-9]*//' -e 's/-(eu-central1|us-west)//')"
if [[ -f "keep/$dest" ]]
then
mv "$file" "research/$file"
else
mv "$file" "keep/$dest"
fi
done
.
├── keep
│ └── name.ext
└── research
├── 001name-us-west.ext
├── 002name-eu-central1.ext
└── 003name-eu-central1.ext
No worries, I get it that this example might be a bit contrived and respect the demands engineers have on our time. The reason I went into such detail was to illuminate potential benefits of "exercising mental muscles" as it relates to knocking out similar solutions quicker each time they are needed.
More seriously rename (sometimes rename.ul) is built to do (some) of this task, and you may have some others depending how "featureful" your install is, but why bother for a one time task when it's not going to be more work to figure out what tool to use, what options it needs, and what parts will still need to be done after anyways?
Honestly this is just a great task for a script, even if you were going to hand write it. It'll be clearer, rely on fewer assumptions, and be more flexible if the task needs to change as you try to do it and notice something else.
Anything more than one query/line of text and one click is more than it would take to get a llm to write this.
The real complicated "bugs" often come from unclear requirements and the hard part is clearing up the requirements. It's more about design than logic errors in the code. And LLMs suck hard at this.
I think "often" is the key word here. To be clear, they often fail for me to! But they also often work.
Depending on your fault tolerance and timeline, one will be better than the other. If you have low tolerance for faults, ChatGPT is bad, but if you are on a crunch and decide its OK to be confidently incorrect some small percentage of the time, then ChatGPT is a great tool.
Most industry software jobs, at least the high paying ones, are generally low fault tolerant and that's why ChatGPT is not entirely replacing anyone yet.
So, even in your example, and even if you write the code all yourself, there is still a risk that you are operating above your own competence level, do exactly as ChatGPT instructs, and then it fails miserably down the line because ChatGPT provided a set of steps that an expert would have seen the flaws in.
This is the big problem with Google’s AI results: before, the wrong answer was from seoscum.com and people would learn to ignore them. Now the wrong answer is given Google’s corporate reputation and also there’s no way to conditionally distrust it so you learn not to trust them for anything.
That's not the point of Google. It gives you a start to research the answer you need. ChatGPT just gives you an answer that might not be correct. So how do you define "successfully answered"?
In programming there are always tradeoffs. It's not about picking the 1st answer that looks like it "runs".
Hold up, if you don't know a language's syntax, how can you verify that the answer returned by LLM is correct (at a glance, because a) nobody writes exhaustive tests, LLMs included, and b) you wouldn't be able to read the tests to confirm their validity either)?
I struggle to think of a case where explaining a task to an LLM in a natural language is somehow faster than writing it yourself, specifically in the case where you know a programming language and related libs to accomplish the task, implying non-zero ROI on learning these things.
It's not really. You could have done the same using no code tools in a similar time.
Question is, would you? It's alarming the amount of trust/praise that goes into "AI".
> if you're an experienced engineer > it took me 5 minutes with AI
You'd at least read the code and fix things up so it wouldn't be 5 minutes.
Adapt to? I'm not saying to not use AI but whether you use AI, do it manually or outsource it it's never 5 minutes. It's about being responsible. Double check what happens. Review it.