To AI or not to AI(antropia.studio) |
To AI or not to AI(antropia.studio) |
I am not a coder (much), but I have to wonder if my experience is common in the coding world? I guess if you are writing code against the version that was the bulk of its training then you wouldn't face this specific issue. Maybe there are ways to avoid this (and others) pitfall with prompting? As it is, I do not see at all how LLMs could really save time on programming tasks without also costing more time dealing with its quirks.
I would like an AI expert to weigh in on this point. I run into this a lot. It seems that LLMs, being language models and all, don't actually understand what i'm asking. Whenever i dive into the math, superficially, it kind of makes sense why they don't. But it also seems like transformers or some secret sauce is code up specially for tasks like counting letters in a word so that AI doesn't seem embarrassing. Am I missing something?
It "knows" what rsync is because it has a lot of material about rsync in the training data. However it has no idea about that particular version because it doesn't have much training data where the actual version is stated, and differences are elaborated.
What would probably produce a much better result if you included the man page for the specific version you have on your system. Then you're not relying on the model having "memorized" the relationship relationships among the specific tokens you are trying to get the model to focus on, instead just passing it all in as part of the input sequence to be completed.
It is absolutely astounding that LLMs work at all, but they're not magic, and some understanding of how they actually work can be helpful when it comes to using them effectively.
I am not an expert but i assume if there are any basic knowledge of the tool then it will try to use it as it knows chunks of some old version. And it wont likely decide to search for the newest docs, you have to tell it search for exact version docs, or feed the exact version docs to context
It is, yes. Surely someone will come and tell you it doesn’t happen to them, but all that tells you is that it ostensibly isn’t universal, but still common enough you’ll find no end of complaints.
> Maybe there are ways to avoid this (and others) pitfall with prompting?
Prompting can’t help you with things not in the training set. For many languages, all LLMs absolutely suck. Even for simple CLI tools, telling an LLM you are on macOS or using the BSD version may not be enough to get them to stop giving you the GNU flags. Furthermore, the rsync change in macOS is fairly recent so there’s even fewer data online on it.
https://derflounder.wordpress.com/2025/04/06/rsync-replaced-...
> As it is, I do not see at all how LLMs could really save time on programming tasks without also costing more time dealing with its quirks.
And that’s the best case scenario. It also happens that people blindly commit LLM code and introduce bugs and security flaw they cannot understand or fix.
Here's a command to copy the man page to the clipboard than you can immediately paste into aistudio (on a Mac):
man rsync | col -b | pbcopy
Not sure about Codex, but in Claude Code you can run commands. So instead of letting it freestyle / guess, do a:
`! man rsync` or `! rsync --help`
This puts the output into context.
Javascript is a nightmare as they change everything constantly. PHP has backwards compatibility for everything so its not really an issue.
It also gives out dated info on salesforce, and im not just talking about the latest and greatest, it recommends stuff that was deprecated years ago.
With an LLM, I can get it to tell me what each parameter it suggests actually does and then I can ask it questions about that to further my understanding. That is a massive leg up over the knowledge spaghetti approach...
Big thanks to all the doc writers who include vignettes right in the documents.
Try N times, adding more context about the environment and error messages along the way. If it doesn't work after those, try other models (Claude, Gemini, ...). If none of those work on whatever number of attempts you've chosen, then LLMs won't be able to help you well enough and you should save yourself some time and look elsewhere.
A good starting point is trying for 10-20 minutes, after which point an LLM might actually become slower than you going the old fashioned way of digging into docs and reading forum posts and such. There are also problems that are a bit too complex for LLMs as well and they'd just take you in circles no matter for how long you try.
Letting some detached LLM fumble around for an hour is never the right way to go, and inversely sifting through the man page of rsync or fmmpeg or (God forbid) jq to figure out some arcane syntax isn't exactly a great use of anyone's time either, all things considered.
Where I find AI most useful is getting it to do tasks I already know how to do, but would take time.
If you understand the problem you are trying to solve well enough to explain it to the LLM, you can get good results, you can also eyeball the outputted code and know right away if it's what you are after.
Getting it to do things you don't know how to do is where it goes off the rails IMO
Is it me or does it feels like the genie in the bottle thing. Remember a TV show where the guy and his friend sat down with the Genie like a lawyer to make sure every angle is covered (going to spare you the details here). That is what it feels like interacting to a LLM sometimes.
(Really the genie is closer to the traditional sci-fi AI in that it's legalistic and rules-bound; the LLM very much isn't.)
Now I dedicate at least one session to just writing a spec file, and have it ask me clarifying questions on my requirements and based on what it finds in the codebase and online. I ask it to also break down the implementation plan in phases with a checklist for each phase.
I then start at least one new session per phase and make sure to nail down that phase before continuing.
The nice thing is if it gets annoying to vibe code it, I or someone on my team can just use the spec to implement things.
I used Gemini AI studio for this and I was very pleased at the result and decided to open source it. I have completely captured and documented the development transcript. Personally it has give me considerable productivity boost. My only irritation was the unnecessarily over politeness that AI adopts in My take is
AI yields good ROI when you know exactly what you want at the end of the process and when you want to compare and contrast decision choices during the process.
I have used it for all artifacts of the project: - Core code base - Test cases - Build scripts - Documentation - Sample apps - Utilities
Transcript - https://gingerhome.github.io/gingee-docs/docs/ai-transcript/... Project - https://github.com/gingerhome/gingee
Indeed, self-invented abstractions are a bridge too far for AI.
You have to keep it close to the path already walked before by thousands of developers.
This makes AI more of a search engine on steroids than anything else.
The key win is skipping the prompt refinement loop, which is (A) tedious and time-consuming, and (B) debilitating in the long run.
I wonder whether LLMs are capable of doing more; probably, we need another paradigm for that; still, they are very, very useful when used right
I don't see how that is a question. I come up with new ideas to improve the LLM-based tools I'm using at least once a day, and the vast majority of these are plain engineering changes that I could do on my own if I wanted to put the effort into it. I think that even if God comes down from heaven to prevent us from further training the LLMs themselves (if God is listening to Yudkowsky's prayers), then we would still have a good few decades of extensively improving the capabilities of LLM-based tools to extract a massive amount of further productivity by just building better agentic wrappers and pipelines, applying proper software development and QA methodology.
The proceeding without clarifying or asking questions thing really grinds my gears.
> Our marketing director (that’d be me) said that if we don’t write something about it, we will be left behind...
Write when you have something to say. What was I supposed to learn here?
Here's the entire prompt:
I need the rsync command to copy local files from `/foo/bar` to `~/baz/qux` on my `user@example.com` server.
Exclude macOS cruft like `.DS_Store`, etc. Here's the man page:
<paste man page for your rsync that you copied earlier, see above>All of this is an attempt at circumventing RTFM because you’re privileged enough to afford it.
Just lay yourself down on the WALL-E floating bed and give up already.
I am backing up and verifying critical data here. This is not a task that should be taken lightly. And as I learned, it is not a task that one can rely on an LLM for.
And if you have a compelling thesis, why hasn't this spread to the investing community?
Well, because nothing in the transformer architecture supports such learning ability. All AI researchers and most serious AI users are aware of this, so I'm not sure I understand the question.
> And if you have a compelling thesis, why hasn't this spread to the investing community?
The investing community believes that they can make money. That's feels pretty much orthogonal to whether the metaphorical intern can learn, and much more related to whether clients can be made to buy the product, one way or another.
Where do you think the LLM is getting its data from? At least on the website you can see the surrounding discussion and have a chance of finding someone with the same problem as you, or learning something tangential which will help you further down the line.
> With an LLM, I can get it to tell me what each parameter it suggests actually does and then I can ask it questions about that to further my understanding.
If it’s giving you wrong flags, why do you assume the explanations it gives you are accurate? LLMs can make those up just as well.
What you should do is verify the given flags on the man page. Not only will it clarify if they exist, it will also clarify if they’re what you’re looking for, and will likely even point to other relevant options.
So, instead you ask a magic robot to recite a fuzzily 'remembered' version of those websites?
Bear in mind that an LLM does not _know_ anything, and that, despite some recent marketing, they are not 'reasoning'.
I just needed two commands: one to mirror a folder from one drive to another, updating only the changes (excluding all of the hidden MacOS cruft). And another command to do a deep validation of the copy. These have to be two of the most commonly used commands right???
In the end, I felt that messing around with precious data without being 100% certain of what I am doing just wasn't worth it so I got a GUI app that was intuitive.
Here is the man page entry for the --delete flag:
--delete is used. This option is mutually exclusive with --delete-during, --delete-delay, and --delete-after.
Hilarious!
Reading and understanding the rsync command would take much more than 10 mins and I am not a total newb here.
A search query takes a matter of seconds to type in, select a result and read. No doubt still under 10 minutes.
But still to my original point it’s insanely more expensive to have chatgpt look it up. This doesn’t bother you because you are privileged enough to waste money there. If time is money then IMO the only valuable time I have with my money is when it’s gaining interest and not being spent.
You can abstract away all the “but I had to scroll down the page and click a different result” steps as “time savings” all you want, but no one was wasting a ton of time there for already well established tools. That is a deluded myth.
I’m not sure I even grasped your point. The delete flag is pretty self explanatory and gives you options for more granularity. Why does that take greater than 10 mins? What is the issue with that entry?
Here is what I get when I type `man rsync`:
``` --delete This tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory's contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files' parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).
Prior to rsync 2.6.7, this option would have no effect unless
--recursive was enabled. Beginning with 2.6.7, deletions will
also occur when --dirs (-d) is enabled, but only for directories
whose contents are being copied.
This option can be dangerous if used incorrectly! It is a very
good idea to first try a run using the --dry-run (-n) option to
see what files are going to be deleted.
If the sending side detects any I/O errors, then the deletion of
any files at the destination will be automatically disabled.
This is to prevent temporary filesystem failures (such as NFS
errors) on the sending side from causing a massive deletion of
files on the destination. You can override this with the
--ignore-errors option.
The --delete option may be combined with one of the --delete-
WHEN options without conflict, as well as --delete-excluded.
However, if none of the --delete-WHEN options are specified,
rsync will choose the --delete-during algorithm when talking to
rsync 3.0.0 or newer, or the --delete-before algorithm when
talking to an older rsync. See also --delete-delay and
--delete-after.Later in the man page, it gives examples and totally fails to explain those examples. And yes, for someone who is going to be doing this frequently and professionally. They should understand this deeply and spending the hours required to be fluent in a command with a kitchen sink full of parameters. I, on the other hand, will be executing these commands maybe a few times in a year.
The more I think about it, the more I think the solution here is to use LLM‘s to write better documentation with lenses for different types of users with different needs.