Why does it look like LLMs consistently overestimate implementation time? I have my suspicion: they estimate how long people would have taken to implement some feature, becasue they were trained on such data. I consistently see estimates of 2 week/3 weeks or 5 days, etc. But then implementation takes a day or 2 max using agents within Claude/GPT. Unless I am missing something? Anybody else notice this? |
We have a tendency to give overly optimistic estimates, best case scenario, no other tasks, no roadblocks...
Whenever asked for an estimate, think how long it would take you to make it and multiply by 5.
I've frequently seen tasks that it thinks will take weeks being done in under an hour. And it will often recommend doing X instead of Y because X requires so much extra work. Basically I just remind it that it is an LLM.
If it worries something is error prone, I ask it to write tools to verify it.
you can blame everything on wired quirks in the training (claude overusing the words "true" and "genuine" when their not needed, AIs using em-dashes because the pretrain has a ton of them)
The actual coding part compressed dramatically.
The unpredictable part is still everything around the code.
Probably big model providers should do calibratuons for that and add an estimation skill.