The assertion is "growing the economy does not benefit the average person; the rich take all the gains". Well, let's think about moving in the opposite direction: "Shrinking the economy does not hurt the average person; the rich will take all the damage".

When I put it that way, the statement seems (at least to me) much less believable. So we have a statement about the shape of the productivity-vs-personal-well-being graph that seems believable if we move one direction, but seems unbelievable if we move the other direction. That makes me much more skeptical of the whole idea.

I think it's more likely to be the case that we aren't recognizing all the ways that productivity is enhancing the average person's well-being.

This is sloppily argued, but it's essentially right about a lot of things. (You can hear the same arguments, albeit made with much more clarity, 150 years ago in Henry George's "Progress and Poverty.")