We need to clarify what we’re looking for. For some people, that’s general intelligence i.e. the ability to substitute an artificial entity for a human entity for all functional intents and purposes with the same or greater faith in their ability to perform a “task” really well (or perfectly if possible), with an absolute minimum of unreasonable errors. This seems like a tractable problem, at least within certain broad domains, if not everything.

Another thing that people are looking for is (take your pick) consciousness/proof that there’s a “someone” there (not just a something)/evidence of an ability to truly create something novel (not just extrapolate) - whether that’s in the arts or sciences. These things are all FAR less amenable to solution because we fundamentally don’t have universally agreed definitions for any of these.

The questions are in the fuzzy overlaps. Can you encode ethics or morality into logic and reason? (One could argue that if the answer is yes, why the heck haven’t we done it yet?)

What do we do when we achieve the first kind of AGI in an autonomous entity with agency in the world, but can’t agree on the second kind? Have we created idiot savants that are useful for most things but responsible for nothing?

AI evals are becoming intractable for the same reasons that measuring human general intelligence has been intractable for 150 years: the constructs resist decomposition, the benchmarks saturate, and the goalposts move. The reasons are the same because the thing being measured is the same. The difficulty is the evidence.