I have a web application that has gotten to a level where I want to start collecting metrics about various parts of the system, specifically to improve various machine learning models I have built and which are currently running. As one example, I would like to be able to record the ML models suggestion and then the actual selection by the user. A lot of times monitoring the interactions is a complex chain process involving multiple steps and stages at various times.

My question is: does anyone who has experience deploying and building / monitoring metrics in production services have any recommendations of software stack and/or best practices? How would you go about collecting the information so you could use it to build better models or answer various questions such as how is a user behaving over time, given condition X, etc.?

many thanks in advance :)