AI might not recursively self improve (part 2)(secondsight.dev) |
AI might not recursively self improve (part 2)(secondsight.dev) |
I get that none of this stuff is like solid, but recursive self-improvement feels similar to a perpetual motion machine.
Base assumptions about the fundamental nature of the world have changed in the past (I like some of the examples in `sapiens`, even arguing that the idea of being able to improve our environment as a species was not believed for most of history) so I could be wrong, but fast takeoff feels like a fantasy.
I also think about cookie clicker math. A new technological paradigm may give us exponential improvement, but we eventually hit the next exponential. i.e. when you buy a big upgrade in cookie clicker, it can start making your current exponential (say if you're at millions of cookies) really fast, the wheel spins really fast until you hit about 1 billion in which case the wheel slows down to a crawl.