How are you organizing your data for your RAG applications? I've searched all over and have found tons of tutorials about how the tech stack works, but very little about how the data is actually stored. I don't want to just create an application that can give an answer, I want something I can use to evaluate my progress as I improve my prompts and retrievals.

This is the kind of stuff that I think needs to be stored: - Prompt templates (i.e., versioning my prompts) - Final inputs to and outputs from the LLM provider (and associated metadata) - Chunks of all my documents to be used in RAG - The chunks that were retrieved for a given prompt, so that I can evaluate the performance of the retrieval step - Conversations (or chains?) for when there might be multiple requests sent to an LLM for a given "question" - Experiments. This is for the purposes of evaluation. It would associate an experiment ID with a series of inputs/outputs for an evaluation set of questions.

I can't be the first person to hit this issue. I started off with a simple SQLite database with a handful of tables, and now that I'm going to be incorporating RAG into the application (and probably agentic stuff soon), I really want to leverage someone else's learning so I don't rediscover all the same mistakes.