The paper misses a critical point. The A.I. may not self improve itself, but it may self improve based on the real world feedback loop.

Example: A.I. provider delivers bad models -> ai provider eventually gets less resources

Example 2: A.I. Provider delivers good models -> provider gets more usage and eventually more ressources

So, the system does not self improve but it gets the feedback from the rest of the world on how useful it is and also on how it can become more useful.