I was exploring this conceptual architecture for neural networks (transformers/attention) based long-context models, its conceptual but grounded in sound existing research and architecture implementations on specialized hardware like gpu's and tpu's.

Can a we scale up independent shards of (mini) contexts, i.e Sub-global attention blocks or "sub-context experts" that can operate somewhat independently with global composition into a larger global attention as a paradigm for handling extremely long contexts. Context shared, distributed and sharded across chips, that can act as Independent shards of (mini) Contexts.

This could possibly (speculating here) make attention based context sub-quadratic. Its possible (again speculating here) google might have used something like this for having such long context windows.

Evidence points to this: Google's pioneering MoE research (Shazeer, GShard, Switch), advanced TPUs (v4/v5p/Ironwood) with massive HBM & high-bandwidth 3D Torus/OCS Inter-Chip Interconnect (ICI) enabling essential distribution (MoE experts, sequence parallelism like Ring Attention), and TPU pod VRAM capacities aligning with 10M token context needs. Google's Pathways & system optimizations further support possibility of such a distributed, concurrent model.

Share your thoughts on this if its possible, feasible or why it might not work.