Interesting article! I think I get what the concern is, but I think your view of human cognition is going to heavily color what you think the limitations of an LLM are. For example, if you think that broadly speaking, the human learning process (concept acquisition, inference and concept revision, etc) is modeled by modern LLM training, these limitations might apply to humans in the same sense they do to LLMs.

Systematicity is a really powerful concept that can drive a wedge between classical cognition folks and connectionists/associationists, but it has never been clear to me that we actually instantiate systematic cognitive patterns, only that in some arenas we should.