I won't deliberately use AI for programming(mahdikarimi.com) |
I won't deliberately use AI for programming(mahdikarimi.com) |
Instead I pop by ChatGPT and ask it what I need done, the tools I'm using and let it write me an answer that I double-check to see if it matches what I'm looking for. It's definitely saved me a lot of time, I still go read the documentation but I have a much narrower point to start from.
I would never just copy-paste whatever it spits out, I don't trust it to generate code nor will I ever trust it to be fed with intellectual property in the prompt (mine or my employer's) but as a general assist for techniques and tools that I've been familiar with, and which could require some researching I do think it has pretty good value.
For example: If you’re an expert in language A, and you want to do something common in A in language B, you can ask GPT to show you the idiomatic way of doing it in B and it will usually lead you down the right path (or tell you it’s not possible).
"Learning something new" from LLMs is like trying to learn something new from a con artist. Sure, they might actually tell you the truth, but having little knowledge of something is exactly the worst possible state to be in while trying to figure out what's true and what's hallucinated.
So then I just forward that back to the model, or in the very worst case use my own brain to fix the problem the LLM generated.
In either case, I'm still better off than had the LLM not existed, because even an incorrect solution is closer to the correct solution than nothing at all.
I now have something that's close to solving my problem that I wouldn't have been able to come up with solely on my own. It just needs a little tweaking to be correct, which I can muster my way through with a little help from the LLM or Google.
Usually you notice it's broken because it doesn't work.
There may be some advice that seems to work, but has subtle issues. (For example, race conditions.) Having more experience can help in spotting these problems.
Either way, copying code blindly isn't a good idea. In a professional situation, better testing and code review can help.
I also use for quickly writing tiny programs to isolate and test a specific piece of functionality.
Those types feel that the end goal of 'working code' is all that matters. Their definition of 'working' meaning that the functional requirements of the feature are met.
That mindset is an extreme. A developer's job is to solve a problem for someone, not to code, but copy/pasting code without understanding as a consistent behavior is a recipe for eventual collapse of a codebase.
I see Copilot and other LLM code generation as methamphetamine for that sort of programmer. They will feel great, but the situation around them will degrade, those surrounding them will be unhappy, and eventually it will be clear the behavior was unsustainable.
From a different angle, I would feel uneasy giving others access to my proprietary data (code), and also accepting LLM generated code into my companies products. The legal rules don't seem fully sorted.
I guess I'll just add programming to that list and not think about it too much.
use a dishwasher from the last 20 years.
More water efficient, chemical efficient, cleaner since the water gets MUCH hotter.
To me it’s about finding the sweet spot between staying connected to the thing that you’re doing for the purpose of keeping life interesting/fulfilling and utilizing your time wisely.
Driving stick is like shooting manual photos. Not necessary in 2023, but can be a lot more fun than the alternative, and enables levels of control that cannot be found with automated solutions.
Choosing to walk is like choosing to forego fire altogether.
So far GPT-4 has proven to be much more helpful in giving me pointers or straight up answers (that I DO check!) to some of my questions than any other source, tool or entity in my life. I've explored and lightly discussed topics I never knew I was interested in.
In my experience it's completely different.
A junior dev who wants to produce a PDF automatically will have to spend a few hours evaluating all the library options and learning their usage. But the code will certainly be free of syntax errors and will produce not-obviously-wrong output before they send it for code review.
ChatGPT will select a library and know how to use it instantaneously - and yet, might call methods that don't exist. Which any human developer would have caught on the very first test run.
They do have certain things in common though - ChatGPT certainly comments code like a beginner programmer :)
Recently I switched to a different case for my home NAS. The drive caddies had a tiny cutout for a label, like 0.75 inch x 1.5 inch.
I decided to see if Chat GPT could take care of it. I asked for it to write LaTeX that would produce rectangles of that size with a light gray border, with the serial number of each drive in the rectangle.
On the first pass it got it basically right. Rectangles all the right size, right border, and all the right serial numbers.
The rectangles were all touching each other so I asked it to put a half inch between each and it did that. Then I decided I wanted fixed width font, and it accommodated that as well.
Then just for fun I asked it to draw a dinosaur using LaTeX. It advised me that this isn’t a good use of LaTeX and then used shape primitives to draw a plausible looking dinosaur.
It doesn't until it does I think.
When I was learning Go I had Copilot on most of the time. I was amazed at how much progress I could make despite being new to the language. And, when switching to Emacs, I'd notice myself reflexively waiting for Copilot to supply autocompletions that weren't there.
Yet, strangely enough, I can actually write Go code now (with or without Copilot). All the troubleshooting, refactoring, redesigning, bug fixing, etc. that I still had to go through saw to that. (Presumably having programmed already for many years didn't hurt.) Repeated and quick exposure to the relevant idioms supplied by my tooling seems to have helped at least as much as it set me back.
On the lisp (Clojure, Elisp, ...) side of things, I have found Copilot (and ChatGPT) much less helpful, presumably because of the relative paucity of training data. Simple things such as matching parentheses seem to elude these tools, and hallucinations are worse and more frequent. I'm still feeling out how this difference w.r.t VSCode/Copilot and other languages affects my experience and the quality of my output, but it definitely feels different (and, frankly, a little clumsier).
Over time I've learned how helpful different kind of tools are for making software. Learning one's editor well. Writing good (and fast) tests. Using linters and code formatters. Using type systems when available. The new AI tools certainly involve trade-offs, but as I gain experience with them I am becoming convinced that putting them aside will make programming a different, and probably much slower, experience than what most programmers come to experience. (Slow can be good, but probably not what most of us want.)
This sounds exactly like what we know about how children learn language. The recommendation now for adults focuses on a large portion of full immersion in content like tv/movies with no translation. We pick up on patterns and rules from repeated exposure. Maybe this will inform how we teach programming too. I had Copilot on learning Rust and it alternately felt distracting and like cheating so I turned it off, but I think I'll turn it back on based on your experience.
You can use an llm, while learning yourself.
I agree with the take on copilot being a hindrance on using your brain. While learning a new language it was really hard to understand whatever is being put in front of me (as autocomplete) is truly best practice or something tied together very loosely.
The extra time it takes you to google, and learn about strategies on how to code, and patterns, and design. This is what is actually valuable, being able to speak to these things in conversation when defending the code you wrote on a PR etc, it makes a difference.
I recently decided to make my editor much more minimal and even went so far as to disable autocomplete entirely. So far I haven’t noticed a reduction in my productivity and actually find it much easier to think about what I’m writing since I’m not having to correct autocomplete and Copilot all the time. All the extra things popping up on the screen were really distracting and I find that for me, a minimal approach gives me more space to think deeply.
I actually love it for writing error messages. Keeps me in flow better not having to switch to writing Human English every five lines when something fails validation.
Problem is, LLMs are really good at uninsightful, repetitive, boilerplate-y bits of code that shouldn't be there. The moment I get to anything remotely complex or subtle, it either doesn't grasp what I'm doing, needs more coaxing than it's worth, or worst of all introduces subtle bugs. It just incentivises me to write mountains of pointless code instead of stopping and thinking if it even needs to be written in the first place.
What we get from code writing LLMs are exactly the same kind of "improved productivity" we got a decade ago with IDEs autogenerating Java boilerplate. It might help in the short term, but long term it just masks the pain that drives tools and skills getting better, more concise, less repetitive, more insightful.
On the other hand, I'm really looking forward to tools that will be able to figure out how much "insight per line of code" our code has, some sort of "Kolmogorov complexity" but with LLMs. Anything too predictable, anything LLMs can easily fill in is a good code smell signal.
For whatever reason the Google results just aren’t as useful as they used to be. Highly specific queries return very generalized results, and it takes a lot of effort to find something relevant.
With ChatGPT I can ask it how to do just about anything in any language and it nearly always gives me exactly what I need.
Sure, pure learning is enjoyable but also ephemeral. Building/doing = learning, yes, but no need to do everyhting.
Despite the widespread availability of flatpacked IKEA style furniture, there is still the desire for the sheer joy of craftsmanship... and there is still demand for the end product.
I know a lot of people who learn by looking at code and then playing with it to understand it. Generative AI is great for those people - they can quickly get code they can start with.
Different strokes for different folks.
Sounds the same like people who rejected IDEs. Or even syntax highlighting. Yes, those people existed and some might still rawdog xml and java in vi today.
(ps. vi might not necessarily be an ide by itself (though plugins can change that) , but the unix command line in its totality definitely can count as one. Sometimes it's just faster to jump into vi, fix a problem, and then continue. )
- Writes the same code (often verbatim) I was about to write, often multiple lines of it. "Ok, next I'll loop over the rigidbodies and apply the force to the ones in range- oh look the loop is already there"
- Gives me some API calls I didn't know about, much easier than finding a stackoverflow post. There's generally not much enlightenment to be had diving deep into the formatting of java.date or some such - if I just want to format a date and move on, it's awesome being able to type a comment "//format the date as.." and get the code. The IDE will yell at me right away if it gave me something deprecated, and I'll hover over the unfamiliar bits it put there anyway to make sure everything is sensible.
- Solves the small problem I'm looking at in a way I didn't think of, or even alerts me to an error in my thinking frequently enough - "Why is it doing that? Oh yeah, I really should've done that.."
My codebases haven't suffered, and I'm still diligent to keep them maintainable, well-tested, and fundamentally sound.
Overall, it speeds things up a lot and frees mental space for me to think about the big picture more. I generally understand the code being put out just as well as any other code I've written - be honest, you don't read a book on every API call you copy from Stack Overflow. The beauty of a good abstraction is that we generally don't have to.
If anything I think it's helped me become a better programmer over the past year, elevating the mental level of abstraction I work at somewhat.
I can see what you said being a problem for somebody just starting out though. And as the author said (and I found trying to learn Rust with Copilot on) it can even be an annoying distraction in that case.
As an aside, copilot and gpt have certainly degraded lately with hallucinations way up (copilot chat has been extra disappointing lately), maybe to save server costs. Could be why we're seeing more negative sentiment.
I don't, because I don't copy code from SO in the first place. But when I'm using or encountering an API that I'm not terribly familiar with, I absolutely do read the documentation for it. It's part of my continuing education and how I maintain a diverse professional skillset.
Just make a mental experiment and imagine that every comment here praising LLM coding skills is actually about copypasting stuff from SO. Does it not seem alarming to you?
Similarly for searching the Internet. Internet searches are potentially dangerous if you can't tell good answers from bad. But if you treat it as a source of hints that you need to verify yourself, it's a pretty good source of hints.
Many of us have learned quite well that LLM's are unreliable. Someone posts about the problem whenever it comes up. It's something close to conventional wisdom.
It's still surprising the first time you see it hallucinate, though. You have to see it for yourself. Once you've seen it a few times, it isn't so convincing anymore.
To me, the analogy to walking would be like foregoing the computer entirely and using an abacus, or sticks in the sand.
The end result of writing code manually or automatically is still a program made from code that must be maintained. Some people like more control over the code.
The end result of using an automatic or manual transmission is still driving. Some people like more control over the operation of the transmission because of the finer grained control over the output of the car that it enables.
However, those are indeed trivial. What's not trivial is not using APIs that exist for your particular problem; using obsolete patterns; not recognising that a particular approach is a dead end; not recognising potential future problems. All those problems fall in the same bucket of not knowing what you don't know, yet having a (seemingly fine) solution, which is worse than not having a solution, because that would force you to go and figure things out yourself (presumably reading recent documentation, etc).
I recognise that some people approached programming in the same way before LLMs, copying stuff from SO and adjusting till it works without ever understanding what they're doing. I guess LLMs can be seen as a neat extension of that "learning style" (which btw is precisely what OP warns against), but it does seem somewhat dysfunctional.
I've just spent the month of August learning to use Godot 4.0 with C#. Resources for this topic on the internet are scarce, Google only wants to show me the official Godot documentation and YouTube videos (which I don't watch because I don't like to learn in this way).
Sure, I could spend my time reading the official docs cover to cover, but I much prefer to get my hands dirty. I'm not a novice game programmer, I've written my own game engines, used Unity and published games. I've got the underlying knowledge of how Godot might work, but I'm unclear as to how to accomplish specific goals I'm trying to achieve.
And absolutely, I could just put everything into Google, and probably find correct documentation for what I need. Or I could just write "// Interpolate from current position to camera" and have Copilot spit out "transform3d.interpolate_with" along with supporting code specific to the code I have right here, which also references functions I don't even know existed. This way, I'm learning the "API landscape" in a sense.
Godot 4.0 is new enough (and C# used with Godot used so infrequently in online resoures) to the point where Copilot literally can not produce the correct output basically ever. The API changed and everything it outputs is subtly incorrect, either generating code that is now obsolete, or that the code it's generating no longer functions because of the API change.
When this happens, I just dive into the docs as a backup strategy. But never as my first choice.
Is this worse than not having Copilot? I don't think so, but it seems like we probably won't agree on that. I still feel like I've 'learned' Godot (I at least know more now than I did on August 1st). I'm able to synthesize the code more easily on my own thanks to the starting points provided by the LLM. And I'm also writing tutorials on how to use Godot, further solidifying the things the LLM showed me in my own mind.
I'm not saying LLM's are a "one stop shop" for learning. I'm just saying it's another tool. One that I think is a little shortsighted to just completely disregard because it's sometimes incorrect.
The response to my comment above is an example of what I was describing, and I don't mean it as an insult (though I understand if it is taken that way). People don't all treat software development the same.
The 'rush to action' types are more willing to shortcut the thought part of the work that others might. They are more willing to embrace AI codegen, as coding for them is not about reifying understanding.
If I were to adopt AI code generation, I'd still have to read the code to understand, which nullifies some of its value.
I do think AI code generation as a way to start designs could be good. But again, if I need to expose company code to do that, it probably isn't worth it.
I don't agree with the implication that I have a deficit in understanding. I would more say that I focus my understanding where it matters.
For example, in web dev this would be a thorough fundamental understanding of the DOM and what interacts with it (JS/HTML/CSS, especially the warts of JS), the performance/responsiveness costs of doing things certain ways, what runs synchronously and how to use that to make the best user experience (avoid flashing/repainting with a quick synchronous thing but longer things should be async..), the overarching architecture of your codebase and how modules should fit into it, what your internal API's do..
If you're rock solid on all of that, the really important stuff of your domain, then you'll have smooth sailing whatever you do. The rest is quite often incidental. Copilot/stackoverflow/some guy on IRC says org.apache.util3.json.JSON.IsValidJson will validate my json, and I already have org.apache.util3 installed? Great, moving on to the rest of the work I have to do building this service..
If that API call blows up on me later, I'll investigate deeper, but my sense of what's important to investigate deeply rarely fails me at this point in my career. The org.apache.util3 probably have little interesting to teach me, but when Firefox started supporting raw JS modules, you bet I thoroughly looked into that.
Personally, I've found that very often, things I've learned that were apparently unrelated to the task at hand gives me a key insight in how to approach the task at hand in a better way. That broad knowledge base is a large part of the value I bring to the job. This continuing education is not limited to technological topics -- any knowledge from any domain may be useful to me as a dev.
This is why I go out of my way to learn things as deeply as is practical, even (or especially) if that knowledge is much more than what I need for the immediate task at hand. I'm taking a natural opportunity to advance my overall knowledge and skillset. The immediate task at hand is only part of what I'm being paid to do. Another part is expanding my knowledge base. A huge part of what I want out of any job is to expand my education.
Other devs don't have that as their goal, and there's nothing at all wrong with that. It's just a different style and different overall goals, is all.
I was describing two different general approaches people use to write software.
Sorry to use your post as an example. I don't know anything about you outside your posts. Please don't take it personally.