Ask HN: Why can't we have LLMs writing documentation?

2 points by simssousa15 1 year ago | 9 comments

Everyone says documenting code is important yet no one seems to want to take the time to write such documentation. My question is wouldn't it be possible to just let a LLM keep reading your repo and generate the necessary documentation? People already use LLMs to code and are trying to make LLMs work as full developers. If we expect them to work as independent developers in the near future, can't we get them to at least write useful documentation first?

jfengel 1 year ago |

I'd break things down into two broad kind of documentation:

1. Comments on code

2. User-facing manuals

There's no need for an LLM to write the former. Comments are left by a person to explain decisions when they're no longer present. LLMs don't go away. You can always ask the LLM why the code was written that way.

As for users... they famously don't read. UX designers take that into account, and try to make their interfaces self-documenting. A user who can't figure it out from the design in front of them probably won't find the right part of the manual to read, either. They'd rather dive in than read cover-to-cover.

The "for dummies" books seem to sell well. I wonder if you could get an LLM to write those.

maxbaines 1 year ago |

This is the first thing I do when using LLM's to create code. So we can and I do have LLMs writing documentation.

In my domain IIoT, I even created a yaml structure to make this easy. https://unsframework.com (wip)

beardyw 1 year ago |

Writing documentation after coding should strongly feature why things were done the way they were. How could an LLM do that?

simssousa15 1 year ago | |

Very true. Generated docs will never be able to do that part.

That being said from what I've seen at work (and I think other people experience the same), the bar is not that high. Meaning if the LLM generated documentation was able to explain what is happening without the "why" that would already be a significant improvement.

taraindara 1 year ago | | |

I’d imagine it wouldn’t be hard to have a kind of hybrid method. The developer comments the code for a reason, and ai can expound on that comment if necessary.

dave4420 1 year ago |

I expect the LLM would need more context to make the documentation useful for readers.

austin-cheney 1 year ago |

Good documentation is no different than good software architecture. Neither are challenging yet most people in the work force cannot do either effectively, because these things require extensive practice making original organizational decisions. Practice isn’t something that can be wished away by some ineffective hallucination tool.

If you think that lack of experience can be replaced by AI then good luck. Microsoft has very recently lost all faith in AI to do any of this for reason.

simssousa15 1 year ago | |

What do you mean by Microsoft has lost all faith in AI to do any of this?

austin-cheney 1 year ago | | |

Microsoft has been investing billions into OpenAI and new data centers specifically for AI. Recently that ended. Microsoft has seen no return from the investment and simply concluded AI isn’t worth it and is not the future of software.