Software support for that kind of usage, when available, will be immensely impactful.

That said, I am more interested in what size model this could manage while producing “just enough” tokens per second to work at a conversational rate. What are models in that class capable of doing for me?

I love "trivial" examples and everything you've said is tue.

In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.

What is this based on? Every researcher I've heard talk about this says it's exactly not true, as an uncontested rule, because the larger models will more effectively contain the smaller models, and use them together in ways that the connections between the smaller models can't. Remember, even MOE is to save compute/memory, not to help performance/parameter.