Ask HN: Are there any Gen-AI products in the wild? I went to a major conference and the AI hype was mad. All devs will be replaced, all testers will be replaced, code will be generated from a prompt each time, tests will be generated from a prompt each time. Utterly absurd, in my cynical mind. But, playing devils advocate, are there any popular tools or apps in the wild that are written in this way? Or even partly in this way? |
The results have been dismal. Fine-tuning prompts is a good exercise for requirements gathering, complete with the tedium that we all love. Sadly none of the AIs that I used were capable of remembering the current requirements with any degree of confidence. ChatGPT4 does a pretty good job until it forgets everything randomly. Both it and CoPilot failed to remember simple instructions like "Please don't send the updated requirements until I ask for them." I assume so that I can waste tokens?
For code creation, both AIs consistently failed to include features that were in the requirements and in pseudo code that they provided and that I approved.
I did enjoy the change of pace of prompt engineering. It's fun while it's happening and while the AI behaves. But it gets very old very quickly saying things like:
Me: We just added <some requirement>, but I don't see it in the most recent version of the requirements.
It: Sorry Dave, you're right. Here's the latest requirements:
<Requirements still without the one in question.>
Hallucinations abound, not just adding things at random, but forgetting things, sometimes with a great deal of resistance to incorporating the forgotten thing.
Also, once you do get to code, don't expect to fine tune it. If you see a bug or oversight in AI-provided code and point it out, it won't correct the code. Instead it appears to re-generate it from scratch. Will there be new, unasked for features? Could be. What about the correction -- is it fixed? Maybe. And maybe in a completely new way.
I have found some success with CoPilot and basic research. In essence it's a smarter search engine. So prompts like this can result in useful leads:
I've got some data that has <a description of some feature in the data>.
Questions:
What is the correct term for <some feature>?
What python library would you suggest to help me analyze <the feature>?
It's quite possible that my prompt-fu is simply weak and ignorant. And using COTS free-ish AIs I'm getting what I pay for. But I would say that as far as using an AI as an all-purpose junior goes, we're not there yet.