How important is it for data scientists to have clean, modular, reusable code? Here’s my problem: while working on a project, I’ll start off in Jupyter notebooks, toying around with the data, doing some EDA, etc. Eventually I’ll pull out some of that code into functions in a Python file, and call those functions from the Jupyter. Neat.

The problem is, as I get more and more functions, I want to organize them more, make them more generalizable and consistent, etc. I’ll also get carried away with organizing files and source control, cleaning up my notes, and making documentation to explain what models/data/source files/results exist, what they mean, etc.

And then I realize I’ve been spending less and less time getting results, and more on this “overhead”. I struggle to balance the desire the rush ahead and get results with the compulsion to make the code “beautiful” and to have the project in the cleanest possible state. I’ve seen plenty of other projects with terrible organization, no documentation, and confusing, poorly formatted code. But if I’m not producing value, my neatness doesn’t matter.

All in all, I’m feeling pretty unproductive because of these habits. Any advice?