Ask HN: How do you manage the drift between implemented code and documentation

12 points by Sheeny96 2 years ago | 13 comments

I work at a medium size financial enterprise. Something that's been cropping up a fair bit recently is that the documentation of processes (how a certain payment system works, what checks it goes through, if this then that etc) have been drifting from it's definition in Confluence (I know, the worst documentation system on the internet, but we're not going to move away from it) and the actual implementation of the code/features.

Typically, the initial document is detailed and well laid out, but it is as we build on it that it quickly becomes stale.

Often, we're relying on one or two particular people with very strong domain knowledge who have been here since the inception of the system to do our checks and balances and work out what we really need to implement.

This is a problem I've seen in a number of places, but it's particularly cumbersome here. How have you managed it?

lasereyes136 2 years ago |

All work has a cost including documentation. You either make updated documentation an explicit work item or it will, at best, get done ad hoc, at worst, it will not be done. If you allow urgency of code requests to bypass a documentation requirement expect documentation to fall behind. If you expect the team to create documentation without explicit time to do so, expect it to not get done.

It is either a priority to do with time set aside to do it or it isn't, and you get what you get. Even a time-box, best effort documentation effort is going to have a better result than "hoping" it gets done and making it a lower priority.

ponyous 2 years ago |

If you have pull request templates, add a checklist item: "Update documentation".

Reviewers should check the docs and review it as part of pull requests. Don't aim for perfect, aim for good enough and updated. Keep it high level, code is the low level documentation.

Encourage devs to write more details in documentation instead of pull requests and link the docs instead.

How do you get people to be diligent is whole other story...

tacostakohashi 2 years ago |

My take on this is that documentation should live in the same repo as the code it relates to.

Depending on the language, you might want to "build" the documentation static html or PDF from markdown or javadoc or whatever, as part of the normal build process.

That way, documentation lives under the same versioning regime as the code. Make a branch to change the behavior... document it in the same branch, merge it together. Check out a version from 3 years ago... you get the matching documentation for 3 years at a time.

Any other "out of band" / wiki based system will always inevitably get out of date, or have awkward references to the "old" and "new" behavior that only loosely correspond to specific versions.

mlhpdx 2 years ago |

Use GitHub hooks, and for each change in the source code identify the relevant documentation (you have to have some way of doing this, probably by convention or a database) and then send the code and the documentation to an LLM with a prompt the boils down to “is this documentation correct, if not suggest corrections.”

Now you have a todo list and a head start on the changes.

This should be a product.

muzani 2 years ago |

Internally (same code), it probably shouldn't be documented. Most of the people should be smart enough to navigate and learn the features backwards from it. If they can't, AI is already smart enough to help.

The cost of keeping docs up to date is higher than navigating it, and as IDEs, paradigms, and AI improves, the cost of understanding code goes down. This also forces people to write cleaner code.

The more docs you have, the harder it is to read it! So minimalism helps.

Some say we should document APIs at least, but even as front end, we figure out what the API does by reading the code. It's easier to read and search the json file with the APIs than Swagger.

However, one thing you should document is common mistakes. Why is it crashing on prod build but not dev? (A: minification) Why does this link not work on chrome (A: chrome isn't supported).

These are the things that work as intended, so you can't just make a ticket and fix the build.

And as others have said, tests are the best kind of doc.

hitchstory 2 years ago |

I built a testing and documentation framework to solve precisely this problem (google for my username to find it).

You write the StrictYAML based tests structured according to your domain language/interactions and then you write one class that interprets them to execute the test and one template file that you apply to generate the markdown/HTML/whatever docs (with test artifacts like screenshots if you want them).

How to documentation then becomes a build artefact that, since it is generated from a test, never goes out of date by definition.

I usually combine this kind of docs with other types (e.g. explanatory) which can't be generated but also dont tend to go out of date.

tmaly 2 years ago |

I have to deal with this same issue on a daily basis. I agree with others that you have to make the documentation task part of the process.

I have Confluence templates for business requirement documents and technical specification documents. These always get jira tasks for any new project.

It is always a challenge to get the business side to write proper requirements. This is where a good business analyst can come in.

In lieu of that, I think having a developer gather requirements in the form of use cases and documenting them with the Gherkin format is tradeoff I will accept. These use cases as acceptance tests can easily sit in a text file along with the code.

kingkongjaffa 2 years ago |

A happy solution is to write more tests, and more 'newbie to the codebase tests' and cookbooks to understand what the code is meant to do or how to use the API layer.

This becomes a test of the use case and can help catch regressions and highlight progressions.

Remembering to write documentation seems hard to get engineers to do.

Whereas having a test pass is still within their wheelhouse of 'I love to code' and can also help with demo'ing new releases as well, since if one of the release requirements is a demo/test case/cookbook entry then you're building in a self documentation requirement.

eftalyurtseven 2 years ago |

We partially fixed this problem with two actions in my current position.

The first one is the Documentation Owner, each person/team is responsible for some of the documentation and must keep it updated.

But we are documenting the features of the product parts. Like the NLP Search Engine and documenting why we are doing this and the project's scope.

for the coding part, I believe the code should not have any documentation it should be lean

codingdave 2 years ago |

It requires a perspective change. The documentation is not the definition of a feature. It was the spec the feature was written to, at the time work began. It is a theoretical snapshot in time of how it was intended to be implemented.

That is helpful for the future, but not definitive.

To help it out, you could add links from the documentation to release notes, PRs, maybe even the code files in the repo, so future readers can see not only what was intended, but the current reality.

interbased 2 years ago |

You could enforce the process so that you only make code updates based on the documentation. So, if a new specification is requested, it must be made explicit in the documentation with a link to the issue next to it. Once the issue is closed, you can remove it from the documentation.

This may be cumbersome to some, but I find it to be effective since you’re also confirming the business logic before implementing it.

jjgreen 2 years ago |

Accept it and make that explicit; make the code clear enough to be self-documenting. Trying to maintain documentation in parallel to a codebase is a fool's errand, it will end up dominating your work and lead to paralysis.

Mockapapella 2 years ago |

tl;dr generate documentation based off of the code automatically.

My solution might be more applicable to back end python dev (not sure about frontend or other languages), but where I work we have 2 systems for generating documentation based off of code.

1. OpenAPI specs. All our code uses the generated clients. 2. I built a system that has a step in CI/CD that checks the docstrings of the functions/methods to make sure it matches the mypy-enforced function signature, along with some other regex-filters.

As for more high level interactions between software (like a graph view of connected pieces of code), you would probably have to use an LSP which could get hairy.

As for the confluence part, there's probably a working terminal interface (3rd party tool, haven't tried any) that would let you upload the result of all that.