I recently released an inference API company (scalattice.com), but the process building it required a huge amount of benchmarking, so we built some internal tools that we felt gave accurate and reliable standards to work off.

Today we've decided to release some of our internal tooling like our benchmark software (locally hosted, open source) you can connect any inference API to the software with your API token, and compare requests concurrently or by themselves, browse the entire model market in one catalog, filter by models and providers.

Building an inference network means you constantly ask two questions: "are we competitive on $/1M?" and "does this endpoint actually feel fast?" We didn't want to build elaborate excel spreadsheets because there are better ways to do this, especially in this new agentic era.

We are releasing it because we think shared tooling helps the whole open-model ecosystem, not just us. If you are shipping inference products or networks, you probably already have a worse version of this in a private repo, so we figured sharing it might help.

Actively looking for developers and engineers interested in building on open model infrastructure with us!

Repo: https://github.com/scalattice/modelmeter Write-up: https://scalattice.com/blog/modelmeter/

Quick Start:

git clone https://github.com/scalattice/modelmeter.git cd modelmeter ./run.sh # → http://127.0.0.1:8765

We look forward to getting opinions on the open source product and feel free to host it yourself. (MIT License)