Understanding Emergent Abilities of Language Models from the Loss Perspective

Understanding Emergent Abilities of Language Models from the Loss Perspective(arxiv.org)

6 points by maccaw 2 years ago | 1 comment

cosmojg 2 years ago |

Does this mean that "overtraining" a midsize LLM for many more epochs on a small, representative subset of the dataset used by a larger, more performant LLM might be sufficient for matching the performance of the larger model?

cosmojg 2 years ago |

Does this mean that "overtraining" a midsize LLM for many more epochs on a small, representative subset of the dataset used by a larger, more performant LLM might be sufficient for matching the performance of the larger model?