I have been thinking about LLMs just like everyone else, I guess! The prevailing sentiment seems to be they are going to change the world, from jobs to politics and everything in between. However, I have been wondering if we are perhaps reaching “peak” LLM and we will see a dramatic reduction in progress.

I am a casual observer of this space so I may be barking up the totally wrong tree, but I thought I would put these ideas out anyway to see if anyone has any thoughts.

Lack of quality training data: Is there a possibility that all the good quality training data has already been used? I would argue that most of the internet doesn’t actually contain good content, and increasing the amount of training data from the internet might actually make these models produce worse output. Moreover, organisations with valuable content could begin to restrict data access, presenting further challenges for training.

AI generated content polluting training data: This is somewhat related to the previous point, but is there a risk that with LLMs generating so much content that the training data becomes polluted with AI generated text. How would this effect the model output? Is it like taking a photocopy of a photocopy over and over again?

Compute Resources: We continually hear that Moore’s Law is dead, are we going to start running into compute/memory issues? Without dramatic increases in both compute/memory are we going to hit scaling issues where it simply takes too long or we don’t have the memory to train larger models?

Architecture Limitations: From my understanding so far adding more and more parameters to a transformer increases its performance. Are we sure that this scales? Or at some point does this performance increase stop or even perhaps go into reverse?

Hopefully these points are not totally off the wall!