Grieving the loss of details(purplesyringa.moe) |
Grieving the loss of details(purplesyringa.moe) |
LLMs are becoming remarkably competent at both of these skills. Not quite at expert levels yet but I expect it won't be long until this happens. I'm astounded at what they can do already, with the benefit of relentless, persistent effort on top of this.
How I've been using LLMs is twofold. Firstly, using chat mode to the effect of having technical documentation to converse with. Secondly, using agentic mode to develop tooling to help me with reverse engineering. This includes things like one-off scripts to analyse complex trees of structures, to modules that use the API of a framework I and others have developed, to prototype how we might want to extend it for specific features. I then write the final code myself in my own style, using the LLM prototype as a rough reference.
For now I think I've got a decent balance between using the LLM as a useful timesaving tool and continuing to understand the finer details for myself. I feel I have to draw a line at this point regardless of how competent the LLM becomes, otherwise I'll just be babysitting a black box, which almost anyone can do.
Isn't this whole post seeking...recognition? I am so confused at the metahypocrisy here. Why is it valid to want recognition for something when you do it, but not when someone who uses an LLM does it?
Training a year, and running a marathon feels more worthy of recognition than driving 42km in a car from the start to the finish.
While I don't empathize with the defeatist tone of the post. I do empathize with this part.
ex: getting a big seed round from a VC for your start up still may show that your parents had connections.
but marathons have this grindability and somewhat stable feedback loop.
cool to see it come full circle and see another person thrown into the industry by the work of another forum member.
I left software and went into violin making and I couldn't be happier (though of course I'm extremely fortunate to have saved up enough in my software career to comfortably make the transition). In violin making a tenth of a millimeter is considered a lot and we endlessly stress over details like the corner shape and the f-holes. And while some of this nitpicking is certainly excessive, it serves more as proof to show that we're extremely careful with the details so that stuff that really matters, like tonal quality and playability, will also get enough detailed focus.
Reminds me of a friend who would go straight to the restrooms for a quick visual inspection upon entering a restaurant: if those aren't clean, there's no reason to think the kitchen is.
The death of every sub culture in a sentence.
A good programmer or a good knowledge worker knows how to work with lossy details.
I was recently replaced by a young developer and the only thing that keeps me smiling is that they still haven’t fixed the part of the application I raised concerns about (because the young dev decided to write on a fresh non-compatible stack even after my warnings). I hope eventually it leads to their own loss of career since that’s what they did to me. All of that to say that Llms have made people into over confident morons.
My manager does not have a software background. Initially he used to come to me with an idea and we used to discuss on how we can technically implement it. He used to value what I had to say. Now he talks to copilot, come up with these grand ideas, dumps all that on our heads in a 30 min meeting and ask what do you think about it?
Offering hobbies, which require ppl to already be supporting themselves, is like a slap in the face.
This is like telling a coal miner that if they enjoyed the work they did in their career, they can go digging tunnels in chalk cliffs after renewable energy destroys the demand for coal.
Said another way: is it impossible to use something which everybody has access to to make something artful?
Or I will try it a third way. Does the fact that people take bad pictures on their iPhones prevent other people from shooting whole professional movies on iPhones?
In answer to that message: taking bad pictures on an iphone is not something celebrate or recognize no.
When an API says it takes something, I need to know the exact form of that "thing". Saying "you can give it this object" doesn't suit me because where are the edge cases, where are the interactions, how can I know for sure I'm giving it what it needs. The docs never tell you where to find the object def, then you spend 3 hours diving into the source code to understand it's just a two item struct or something.
LLMs introduced an explosion of complexity, suddenly code bases sprout out of nowhere, and there's no point in following all the different datapaths through because the guy who loves testing out the latest claude will just change everything in a few days anyway.
With low level programming, nearly everything is just raw data, aligned to meaning. I like to tell juniors confused about file types, that file types are a social construct, they're just a bag of bytes with some structure. Same thing applies with low level programming. You know what a byte is, you know the endianness, the signedness, you can feel comfort in knowing you're not missing anything, and then you build up from there seeing each block build upon the previous.
A compiler is a rather predictable piece of software; reproducible builds are a thing. An LLM has approximate knowledge of many things, and approximate and fuzzy ways to do anything even moderately complex. This is great for research, ok for planning, and it sucks for execution. The fact that LLMs can write satisfactorily working code from high-level requests is a miracle, and, as with most miracles, we're likely not noticing something, being dazzled by the slight previously unseen.
I like lifting heavy things and moving them around. In fact I like it so much it’s a massively effective buoy for my mental health. However, I don’t expect the economy to make space for me to move farm equipment around to make food and expect to get paid a living wage doing it.
Software like food is now a civilizational requirement and transient LLM crappiness notwithstanding, accessing well-built flexible plentiful software is a major unlock for humanity.
I’m happy now to go the gym and lift heavy things and let everyone have plentiful food grown by mechanized agriculture. I’m looking forward to do the same for my intellectual pursuits.
Take, for example, Claude Desktop and Claude Web. They recently bragged about reducing their first paint from 4500ms to 1000ms. For an interface that displays text and sends/receives HTTP chat requests for more text, let's remember. This is from a frontier company that can allocate infinite compute to the frontier models consumers don't even have access to. Let's not get into how it consumes gigabytes of memory. We wrote more efficient GUIs than this in the 80s with 128kb of RAM on a 4mhz CPU.
These despair pieces do not reflect reality in any form, and they're so far from reality that I believe this is more likely to be an article written intentionally to deceive people who don't know better than it is to be genuine. Performance engineer jobs are not going anywhere. LLMs are not generating better compilers nor better kernels than what exists. They're producing insanely inefficient CRUD garbage.
It's still valuable to deeply understand parts of a program, but we don't have any tooling that helps us do that. We just have to raw-dog it by thinking really really hard and remembering how all the code connects together.
I want a tool that gives programmers a place to record their thoughts. Developers need a place to draw and write, and also interleave blocks of code that automatically update to match the actual state of the code.
The closest thing I know to this is org-babel, part of Emacs, which allows you to push code blocks out of an org file into an actual source files, or pull them in from actual source files. This is mostly done manually by invoking functions called `tangle` and `detangle`.
I intend to investigate this further in Emacs, since I'm an Emacs user, but Emacs is never going to be the friendly UI we need to make this tooling common.
LLMs are not very good at performance issues unless you walk into it with a clear idea of the likely root cause and the bigger picture regarding actual hardware and desired customer experiences.
"Please make the code go faster"
vs
"I am noticing what appears to be contention between threads under workload A, B & C, but not with workload D".
These are completely different universes of capability and outcomes.
Even if an LLM can fight its way to the answer on its own, you can achieve a specific desired result much faster and with significantly lower risk if you are genuinely an expert.
I've seen a concrete example of this recently. I profiled the client's product "the hard way" and arrived at a change to a single line of code that would eliminate a mutex issue. One of the client's developers used the LLM and wound up with a change set that touched hundreds of files, but otherwise achieved approximately the same performance fix. The other developer even had my hint that it was a single file change and couldn't figure out how to do this despite prompting a leading edge model regarding this exact possibility over and over.
Taste and aesthetics apply to absolutely everything. Not just UI/UX design. Perhaps it is even more important that we care about the things that are invisible to the customer. It is certainly easier to forget about them or treat them like they don't matter as much.
This isn't about having trouble adapting to change, because many engineers going through this are adapting just fine as far as others can observe, adopting LLMs, changing the way they work and whatnot.
The grief comes from either that new normal being something they don't identify with anymore, or that new normal being just... different. The former will inevitably prolong the grief, while the latter will eventually result in the grief subsiding (even though some sadness for what was lost will never really disappear).
We're being asked to adapt, but perhaps we should just accept that things are not going back to what they were, look around, and decide not to do the things we used to love in a new way that we cannot love. Others might not even notice we chose that path, and think we just "adapted".
Does this make sense?
Instead of cache, software-managed fast and slow memory spaces.
Instead of branch prediction, memory with multiple read ports/buses and software-managed parallel prefetch queues.
Instead of hardware speculative execution, VLIW.
Instead of automatic DRAM refresh, a hard-real-time OS handling the refresh timing.
I don't care that this would to make things on average slower and more expensive. Modern computers have tremendously fast average-case performance, but the software is still mostly laggy garbage. We sacrificed ease of understanding and didn't even get responsive software in return. I believe that if modern computers really were fast PDP-11s, we would not be in the situation we are now, where most programmers see nothing wrong with software taking some unpredictable human perceptible time to respond to inputs, even when simply processing text. I care more about worst-case performance than average-case performance.
As the article says, "Like a fisherman might feel one with a fishing rod, I treat the machine as a continuation of myself." A fishing rod always responds in zero milliseconds. A computer should do the same. You can say AI is qualitatively different because by embracing it we abandon even the possibility of understanding, but in practice we mostly abandoned understanding already. There's no practical way to count cycles any more.
Of course, most computer users don't care about this, so there's no commercial market for the kind of hardware I'm imagining, but I'd love to see the "fast PDP-11" approach taken to its limit. Let's make computers it's possible to completely understand.
While I sympathize with your loss, this was always going to prevent you from holding down a job, even as a pre-LLM "coder." The majority of us have to work on systems, not just single files or cleanly isolated programs we can hold in our head.
I'm less pessimistic than the author though. There is always room for people who know what they are talking about. Take a deep breath.
I’m working intensively with LLMs to find out what they can and cannot do and for all the capabilities in there they remain terrible at compressing functionality into few concepts and as a result they produce incoherent (read Fred Brooks on coherent design!), failure-prone products. In other words: I expect that there will be good demand for people who refuse to let code grow beyond something they can keep in their head, even though the means by which one accomplishes this might be different than what you do now.
Hope that there is some solace in this.
It is tempting to extrapolate the fast progress and conclude that everything will be automated soon, but it still appears to me that LLMs have a “spikey profile”: while very good in some areas, they fail completely in many others, with no evidence that this is only a matter of time.
I'm lucky though - as a hybrid engineer/manager, I've been able to lean away from the former and into the latter.
And I've come to notice that humans doing agentic work seem to need more emotional support per unit effort than those doing traditional engineering.
So there are opportunities there. I don't want to spend a second longer than I have to prompting machines - there is zero dopamine loop in that for me, I get more doing housework and at least that makes my wife happy - but I'm happy to support a few other humans engaged in that kind of work.
It's sad though. I used to love dreaming intricate machines into existence and then watching them come to "life", or at least actually start working. That dream seems dead for now, I hope it comes back but I won't hold my breath.
Over the years, I’ve transitioned from building “beautiful” stuff to nowadays building stuff that works. I still want it to be perfect but now not for me but for the user. If I think it’s ugly, but the user wants it that way, then who am I to decide against it? They will have to use the software, not me. Seeing a user be happy with the software is super nice. Way more enjoyable than me just making stuff for myself.
Thanks for making me feel old
What I think we're likely to witness here, or at least what AI investors ultimately are hoping to see happen, is the displacement of code as we know it (often already sorely lacking in quality and craft) not by more of the same varieties of code, but a massive profusion of shittier, more homogenous code. It won't have to win by being better; it'll be able to do that by being cheaper alone. And we're frankly kidding ourselves if we think that doesn't mean a profound deskilling and potentially deprofessionalization across the whole class.
I agree but it's also better. On average for most programming tasks I think humans are as "defeated" as coders as we are as chess players.
The machines will not only be as good or better than you at the DB design, the business logic, the performance critical algorithms, the UX, the performance tweaks, the accessibility standards, security holes, browser compatibilities, laws and regulations and whatever else you need to make the whole solution.
It will also write user manuals in any language, and rewrite them when needed even on a friday evening. It absolutely will not stop, ever, until you are DE...wait, I mean DONE!
So the situation for coders is even worse than it was for the weavers. The machine delivers not only cheaper and faster, but also better. :-/
If the job involves mostly working on a computer, it will be probably gone in 10-20 years.
I don't think comparing it to the industrial revolution is appropriate, as the political and social conditions were completely different. I think you can compare the disruption that is introduced by artificial intelligence more with the deindustrialization of regions such as the Ruhrgebiet in Germany.
Wrote a post some time ago on how we'd be helped by segmenting and ranking the domains of our systems so we can be deliberate about where we stop short of full automation: https://ljtn.github.io/epiq/blog/cost-of-cognitive-debt.html
It is a giant interpolation machine. It is very good at remixing stuff, but is is hapless when it comes to novel things.
Consider this: an llm was trained on physics, but the training data was cut in 1904. That is just a year before Special Relativity was published. The LLM was then shown papers of SR, GR, and Heisenberg's paper from 1925 that established quantum mechanics, and similar foundational papers of what we call "Modern Physics".
The LLM has systematically "disproved" all of them, and rejected as false.
So even if you show it a novel idea, it's still going to reject it.
It is by design limited to remix existing ideas.
The recent "novel" mathematical proofs are also just remixes. What I mean it uses two or more established math frameworks together to generate a viable bridge. The result is undeniably a new proof, but not a novel idea.
People with novel ideas will always be needed.
I'm trying to reinvent myself now and learn the other sides of the projects; how to monetize, how to promote my projects, how to "finish them", etc. I understand that's also changing radically since many people are trying the same thing, and with LLMs the market is changing in unpredictable ways, but that's also exciting! I can get so many more things done now that before I just didn't have the time or focus to finish.
Ouch. There's an idea that I've seen a few times now that there are two key types of developers: developers who thrive on building products and solving problems through the existence of code, and developers who thrive on the process and the low-level puzzle of getting that code to work and then to work well.
The former are having a really great time right now. The latter are feeling justifiably threatened.
Hard not to see it as tech killing competence in general, and cheering it on.
Kinda wish I'd gone to med school instead. Tech has only shown itself to be more unserious as time has gone on.
If I grant you that cheap plentifull code will be a net boon to humanity, it's still normal to be unhappy that it's you specifically that will need to be ground-down in the cogs of progress. 5 years ago I also believed that the future could (and would!) be vastly better for many people, I just didn't know that betterment would be conditional on me losing the work I enjoy. In that sense LLM's are only a loss to me.
Many people feeling equanimous about this stuff are financially secure, later in their career or working at a manegerial level. That's fine, but at that point you're obviously not suffering the same pain as the author.
There is a lot of the latter happening so my comment remains relevant.
“I need to block abundance for all because I don’t feel safe with change”
is probably the most dangerous sentiment coursing through our civilization right now, and it’s coming now for me, and this is my rationalization from giving in to my baser instincts.
I’m imploring others to do the same
The whole thing was quite functional, and a few thousand lines total, and built a bit naively - the framework and the app were under 10k lines combined total.
Later I tried rebuilding it for Win32, and it ended up being more code (sans bespoke UI framework), and I'm not sure it was better, though I'm certain, the whole mass of code doing the thing was an order of magnitude more.
If I were to attempt building it in Electron, I could add 2-3 orders of magnitude to that easy, all for the same basic functionality.
All I'm saying is abstraction isn't really what it's cracked up to be.
It seems in our profession there might be less satisfaction from work, but if it leaves me with more space to accomplish things in my own time, I think I prefer it.
I know things get worse before they get better - but some of us are at the point in our career where we likely won’t be around to see it get better again.
Those are unfortunate examples. The US doesn't do them very well compared to other OECD countries.
Are you thinking Southeast Asian countries, or European, or what?
Libraries are the last time americans agreed on basic principal that a public space with a public good is worth investing in everywhere.
Why is america so much against public services in general?
Developer tolerance of low performing software has been a problem since before LLMs though. The whole industry seems to think O(seconds) is a perfectly acceptable amount of time to launch a program. For decades, as computers got more and more powerful developers tolerated proportionally poorer and poorer performance. There's almost no such thing as a "performance-oriented human" anymore outside of a few niche industries, and nobody is really producing and sharing much hand-optimized high-performing code anymore, so it's no surprise that LLMs trained on the Internet are not good at it either.
This is, unfortunately, false.
Yes, if you one-shot some code with an LLM, it can be terrible. But if you let it profile and optimize it, there is no obvious limit. LLMs have more patience to investigate performance issues and fix them than humans.
Again, I wish this wasn't so, but I see it in my job daily. There is code in my projects that is vastly faster because of LLM optimizations. Not only can they find more in less time, but they find things I and my co-workers would not have thought of.
You could dream all you want about how you would write better version of claude desktop. Fact is: you didn't. And you never will.
I already did, actually. I work in an LLM startup that produces small models for specialised purposes and I wrote our frontend interfaces, which are vastly superior to both Codex and Claude's interfaces. Unlike OpenAI and Anthropic, we are profitable, turning 8-digit revenue with zero outside investment. We have complete ownership, without tens~hundreds of billions in expenses and debt. Excellence in software still has a place in the world, even if the VC darlings get most of the attention, because at the end of the day consumers and enterprises alike will pay for software that truly works well.
I find forcing it to visualize things immensely helpful. I'm usually studying git diffs but when working of a big feature or refactor that can just be too hard.
I've never been very pro "visual programming" and always hated UML et al, but part of me is starting to wonder if it's time for us to give it another serious go.
I do it differently, I focus on better recording what the user wanted, the so-called "user intent". To do this, I record all messages typed by the user since the start of the project, whether 3,000 or 10,000 messages. An LLM can churn through them in 10 minutes and derive a fresh, up-to-date interpretation from the raw data. This can be used to judge whether the implementation has diverged from the intent, or, in other words, to realign the code and tests. The messages the user writes are usually designs or corrections, a very rich, compact signal. If the user struggles with something, it could result in a tool, a skill, updates to the project docs, or new tests.
Sounds, like you already mention with org-mode or similar ones) like literate programming (https://en.wikipedia.org/wiki/Literate_programming) or jupyter notebook.
I think the solution is still code, just at a much higher level of abstraction. Maybe a start is kind of typed ADR or FSM that guides (constrains) the agents. I believe more type checking guarantees will be more and more important for agents.
Now, when someone sends a working PR in, even high quality and well tested, they may actually have no idea how it works.
Building a basic X11 window manager is almost a one shot prompt.
Modifying a UI toolkit to make it work with MSAA/IA2 is simply not possible.
There's a lot of room for deep work left... for now.
If I were trying to accomplish this particular goal I would first consider what the agent could see. In particular does it have an accessibility inspector of some kind? or even NVDA hooked up with NVDA Remote so that it can actually see the implicit a11y tree for the toolkit it is working on? My email is in my profile and I would love to chat about this.
Part of woe is that once you've reviewed, validated, and comprehended a piece... Later gets casually mangled by some other LLM-generated urgent change.
Could be just defining the methods without filling them but depending on the mood I code more by hand or less.
Which is frankly exhausting to do when you have to keep up with the rate of LLM changes
Or at least that's the current model I'm playing with.
You’re probably limiting the frontier models by being specific.
I do feel there is a limit to LLM-s today. I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".
Knowing where that limit is, is as hard as it always was knowing how fast a team of engineers was going to do a project.
Theres lots of bad Pytorch code, just go ask george hotz. This isnt the magic you think it is. And nobody wants to hire the guy that just points llms at things and says make it faster. Things have value because talented humans make them. Its why a luxury coat is worth more than the linens that make it, or one from walmart made by a machine. This will 1000% apply to programmers. I think OP will be fine.
Pointing a frontier model to a 10 or 100MLoC codebase and saying "make this code fast, make no mistakes" doesn't work. As an experiment I recently tried this with a relative small (500KLoC) codebase and it got stuck on believing that the primary cause of slowdown was the database not using a connection pool. (Which was completely irrelevant for this specific code.)
In general the OP is right, the people who get the most value out of LLMs are veterans.
Very well put.
A large part of that "what comes after" is grappling with the next hard question: after the death of the craft we loved, are we - our skills, our intuition, our problem solving - even needed anymore?
Even if we still are now, will there be a time when there also won't be a place for us?
I would love to take this path but unfortunately I have to make money to pay my bills and afford to eat. I don't know how I would afford to keep my home without continuing this career, as mangled as it is now thanks to AI
But you might be able to choose to leave that to others, and fill some other role in the development process where you don't have to pretend.
It will still be with LLMs, though.
The problem with this outlook (not with you personally) is that the increase in accessibility for you comes at a cost, but the way things work these days the cost is not paid by you but by someone else — someone you'll probably never even meet. The cost has been abstracted away from you and foisted onto somebody else against their will. This shows up as people adversely affected by local data centers (increased pollution, higher electricity prices), people displaced in the workforce (author of the article), people of the future who will not understand things because it's easier to skip understanding for now (students, early learners), and so many more.
It's very liberating — so long as you are given the ability to not think about the consequences for these other people, and the abstraction process by which AI companies are providing their services gives you that freedom by design. At the very least, it is something about which you perhaps ought to be wary.
I saw a video today, talking about how this is really the end goal of what we've been working towards for 200 years. Automation and scale and convenience uber alles. As structured it's not a good idea and it seems that Dr. Kaczynski wasn't wrong in diagnosis, only in the treatment. But here we are and we're pretty pot-committed, so I guess we have to see for ourselves what's on the other side.
That being said, I can't see this entire field existing in five years anymore. I'm hoping for at least two more years, but who knows?
This stuff is coming for all white collar, the barrier to entry is completely gone now. Maybe not the barrier to mastery (yet), but the bottom has fallen out.
I don't care whether I'm writing code or reviewing LLM generated stuff, but the prospect of losing my job and having to survive on welfare for the rest of my life isn't nice.
And I’m wondering if this isn’t the enormous amount of organizational debt from having security second to everything finally coming calling.
I _am_ a craftsman - software, wood, and a few more domains. There is a lot of personal satisfaction I find in woodcraft through the motion and the exercise. I love that this is a luxury hobby instead of a personal necessity. The difference there is that if I take my time on personal necessity where the market isn't paying for it, I may take food out of my kids' mouths or lose the roof over their head. Luxury craft hobbies face only self-imposed pressures.
AI is letting me build similarly. I can continue to craft my Rust and my Python and my Typescript and my Fortran to my heart's content - and those skills help in the day to day - yet I'm also able to compete in the market and build things that were really infeasible before.
How are you keeping your “human” addition to the loop valuable, is it through the time spent on the software craftsman hobby?
Im in my early 30’s, and trying to stay ahead. I feel it was easier prior to LLM’s, and now its tough to even know where to focus skill building.
But what will their day to day look like? Meetings?
Previously, I'd have to work days undisturbed to get important stuff out of the door. There was effort involved to reach an elegant solution that fit business need.
Now I'm a meat bag pressing enter on a "recommended" option Claude already figured out was the best approach.
And how long can that last? We’re expensive meat bags…
If you are constantly thinking claudes approach is the best then perhaps you were not that good of an engineer to start with.
In the short term, I have seen a lot of managers and other higher-ups talk about how we do not need to worry about low-level details anymore. In their minds, we are now all designers and architects, so we do not think about the small implementation details that ultimately do matter for performance and reliability.
People who know what they are talking about worry about all of the details from the big picture down to the small scales. We can still operate using abstractions like designers and architects, but we must know enough to choose the right abstractions that account for the concrete details properly. I've discussed this point earlier this year [1] using the tree swing diagram [2].
I'm very interested in systems as well, but being less sure about the future (on whether this is something I really need to think about, or whether I'll get opportunities to work much at this level), I started reading about such topics a fair bit less.
It doesn't yet know when I brushed my teeth last, what specific foods in what quantities give me heartburn/indigestion, what that funky smell from my running shoes might be.
It can write pretty good code on a recursive loop when its provided a clear target. It can write better code when it has someone who understand architecture guiding it. It can review code reasonably well as well.
ChatGPT tied to robotics might even be able to do more interesting things!
It does pretty poor on highly specific knowledge where a RAG better supports -- something like Agent Search at GCP or AI Search at Cloudflare. But it can synthesize.
In effect, we've built an amazing library registry and need to up our librarian skills and the skills of people or systems that can use the information the librarian and their system can find.
Knowledge has been available just by asking Google for decades now. The LLM makes it easier but it's a difference of degree not of kind
Until the models are 100% reliable knowledge will be required in order to quickly spot issues and work efficiently with the model to address them.
I'm not sure how it plays out in 5 or 10 years, but that's how it is now.
So as market pressure goes, that leaves employers. I do see the occasional oddball project that resists this wave, but "doing things correctly even if it takes longer" is not a value you're really allowed to have in an economy where the make or break factor for your business is usually getting investment capital, and capital is, as a population, probably the most all-in on LLMs of anyone, to the point where I believe they'd push for vibecoding even in instances where they can't find numbers that justify doing so.
A ton of people not in tech hate genAI so much that they say they'll, for example, not purchase a game if they know someone used it for any part of it, and yell about it online, and such, but games are pretty much the only consumer-facing software people make purchasing decisions about, so maybe that moves a needle there, but gamers failed to rally against microtransactions, DLC piecemealing, or even things like always-on DRM or revoking purchases that literally remove their ability to play their games, so I doubt that's going to materialize a change in something that could be more easily obfuscated in response to this pressure like the provenance of the software
Market pressures require at least a somewhat free market, and the overall american market for software is drastically distorted by various oligopsonies that by and large has a vested interest in LLMs (and specifically the corporate black box ones) being used for as much as possible
We have tailors around the world, but Zara and other brands do the lion's share of business.
Automation in physical labour resulted in less work for physical labour. There are still some people producing hand crafted work, but the market doesn't need that many of them.
A much smaller group of people can serve the needs of an entire state.
I’ll give you an example of one that was written recently actually.
The initial compromise happened because the app explicitly did not verify auth claims when a specific string was in the ISS field. Well, fuzzers exist and are common.
The next issue was that once you’re in, there was no delineation between admin and regular users. Everyone had all privileges if they just made the calls.
Anyway, we did the usual post-remediation investigation and write up. The devs were of course using the latest models, as they were instructed, and the issue stemmed from a problem they’d been having integrating a specific company into their auth scheme.
Eventually, after many enumerations, the model opted to just skip auth altogether if that companies ISS was present. The devs, being in the habit of just accepting the changes did so and because of the nature of the code implemented nothing caught it in the pipeline.
This is sadly an incredibly common story and it won’t be fixed by models improving I don’t believe.
Yes, and, integrating it into my normal tasks and delivery. Some principles:
(1) I wouldn't worry so much about staying _ahead_ of the technology, rather, focus on the outcomes for yourself and the people you serve. That gives a more holistic and natural boundary when you research and adopt the technologies that help you get there.
(2) I'd also focus on adopting what works best, not what is latest and greatest. Tech regressions definitely happen[1], and you know your need the best.
(3) When quantifying, cost-benefit is typically focused on Benefit / Costs - 1 to give incremental lift. Unsurprisingly, if benefit is high and costs are higher, it might not be worthwhile to adopt!
[1] https://roderick.dev/writing/2026-08-28-obsessing-harnesses/ forgive a small bit of self promotion, but I'm researching this exact problem with my research group, about how to quantify solum benchmaxxing, tradefoff adoption, regression, and improvement for harnesses.
What seems to happen though is it always gets pulled into a discussion of "But they can't X" or "but ma taste!" or the old generic canard "can't replicate what's not in the training data"
All I want is for people to see that yes, this is happening, accept it, then figure out what a good response would be to it. Instead we get everything from stochastic parrot parrots to "Dario is just marketing when he tries to warn us" to the old an thoughtless "But if you think it's bad, why are you doing it?"
Please.
There are many things about our modern world which make less intuitive sense than the lives of hunter gatherers. People are trying to figure out how to survive in the conditions they are placed in.
Physical labour was weakened and then had to compete with machines, till we got lights out factories.
We were left with the service economy to find roles that allowed us to thrive. Now that is being threatened as well. It is unlikely that LLMs are going to make us all into entrepreneurs and capital owners.
Not to mention, the service sector required far fewer workers than manufacturing.
This isn't quite the same kind of moment we see over and over. It's not simply cars taking out horse drawn carriages.
I do wonder though about the optimizations within it, what could be the most optimal way to achieve such room (ie. knowing which questions to ask)
Yes, learning and tinkering is still really the greatest way to achieve that
but I think what I am talking about can be better simplified with the analogy of a gym: previously what used to be necessary (manual work/labour) but when most people got into information work, even then there was/is a need for it (physical work), then we saw the evolution of machines specifically designed to optimize for it and we got machines specifically designed for this training, which helped push people's body to their absolute limits.
I do wonder if an hyper-optimized environment of learning and for asking questions (or more so knowing the know how on which questions to ask), this whole process might be optimized for it and what that process might would look like is a source of curiosity to me.
A relevant video which talks about similar topics: Bodybuilding for the mind: https://www.youtube.com/watch?v=o0DtxUJ6rAc
But there's no reason for us to be angry about it or resist it, of course
I think you might be viewing AI and humanity as a zero sum experiment. It's really not. Go read some David Brin (Existence is a good start). We don't know all the positive and negative aspects to come, but we aren't in a dark forest situation. Existentially, AI is here, what are we going to do with it?
Working on platforms of sufficient scale, there’s a lot happening that’s out of your control. You can build the mental model around it, but it will contain a lot of black boxes where an abstraction hides details.
Generated code changes this, probably, but I’m curious by what degree.
I felt there were limits 4 years ago. I couldn't get even an 8k context window.
It's starting to feel like the important gaps are the only ones left that need to be closed before there aren't any left. It also feels like next year they will start meaningfully closing.
I do feel there is a limit to human-s today. I did not try, but I doubt it would work if I asked literally any programmer I know "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".
Hell, I'm willing to bet they'd fail at this task even with an unconstrained snacks and kombucha budget. Humans have a long way to go. My job is safe.