I recently interviewed Kyle Corbitt (YC23 Founder) who has been deeply involved in the LLM fine-tuning space the last couple years. Much like with pre-training models, most of the performance gains ultimately delivered from a fine-tuned model occur as a result of well planned and executed training data curation.

I whipped up this article sharing important best practices patterns that have emerged in Kyle's experience observing the fine-tuning of thousands of models across a wide variety of downstream tasks. Some validate long-held understandings in the space, others were quite surprising to me (especially the sample efficiency of modern SOTA LLMs!)

Hope sharing this knowledge helps someone out there! And please share additional insight you have in comments so I can learn even more about this topic.