A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.

I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.

One, things like catastrophic forgetting and falling into incoherence.

Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.

As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.

Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.

However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.

And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.