This is a very cool nontrivial problem. I worked on this some in a mixture of agent type context. There were lots of failure modes mostly involving model aggregators not working well because they tend to end up plagiarizing what they consider the "best" LLM in the target pool (that is, they don't actually give a balanced view of the target results, but more a winner take all). I had to build all sorts of tooling to avoid this to "anonymize" the content from the LLM target models.

And I only went up to 7 LLM models in my swarm. It seems this has way more, so the issues will only be exacerbated (it isn't clear how much improvement you get once you go beyond, say 10, LLMs in a swarm).

I'm curious what mechanisms are being used for aggregating results returned from LLMs. These are really difficult problems.

I play around with OpenRouter Fusion some, and it does a pretty good job: https://openrouter.ai/docs/guides/routing/routers/fusion-rou...

One nice thing with that application is each LLM is mildly agentic, as it has access to the web.