Yep. Where I work there are _a lot_ of efforts underway in that direction. Put simply, standard transformers are great, but they’re very expensive, both to train and to do inference with. They also need enormous datasets. We need architectures that are compute and sample efficient, and friendly to hardware. A standard transformer ticks none of these checkboxes, and research is needed to actually be able to make money with these models. And because profits depend on this research, it’s going to bear fruit. The field is vast and relatively unplowed.

Exactly! Quality can be very important too, as a series of interestingly advanced small models have shown.

I won't say I totally disagree, but I think the article overstates the position.

Two of the most important developments for ANN's were backprop and then the deep network training method. Without those we could only evolve ANN's which is much less efficient than the training methods.

If the scale argument held 100% then teams that purely focused on evolving ANN's (this is the "search" method of creating ANN's) to their target would be in the lead, but I'm not aware of anyone doing that other than smaller/hobby size projects.