Transformer vs. Post-Transformer Debate: Kaiser, Kosowski, Jones, Lechner [video]

Transformer vs. Post-Transformer Debate: Kaiser, Kosowski, Jones, Lechner [video](youtube.com)

5 points by Cappybara12 77 days ago | 3 comments

I found the disagreement striking. Kaiser argues Transformers still win unless someone shows a better scaling curve while the other researchers argue the field is overfitting to current hardware and missing better architectures.

There was a back-and-forth on scaling, hardware constraints, continual learning and latent reasoning.

shivcodesai 77 days ago |

Very Interesting panel, Kaiser is one of the only scientists in the field with no social accounts, no public presence and who appears in the media maybe twice a year. That’s exactly why the full talk matters.

Cappybara12 77 days ago | |

I had the same reaction when I came to know. IMO, the panel is interesting cause Kaiser wasn’t especially dismissive of the Post-Transformer side, in his rebuttal he explicitly said he was “very sympathetic” to their arguments.

He also more or less conceded Adrian’s framing that we still haven’t had a real “PageRank moment for intelligence” yet even while defending Transformers as the strongest thing that currently works and scales on the current hardware.

One of the sharpest lines in the whole debate is probably Llion’s version of the local-minimum argument: Kaiser may be right up until the day a real breakthrough arrives and then wrong forever.