Dispersion loss counteracts embedding condensation in small language models(chenliu-1996.github.io) |
Dispersion loss counteracts embedding condensation in small language models(chenliu-1996.github.io) |
I wonder if anyone has figured out how the information is compressed and calculated the amount of information an LLM can hold depending on its size
You might want to look at Physics of Language Models[1]. IIRC, the authors estimate it to be ~2 bits of factual knowledge per parameter.