We've built a data generation studio that creates robust test datasets for testing their LLM applications in just minutes. In talking to lots of teams building with LLMs, we discovered a critical gap: everyone wants to build AI products, but acquiring, labelling, and organizing test data for tasks is a massive challenge.

The current approaches to creating "golden datasets" for LLM testing are either infrastructure-heavy (observability-based) or time-consuming (manual creation). Many teams end up relying on vibes-based development, which leaves crucial questions unanswered about model selection, edge cases, and performance optimization.

Our solution generates comprehensive, realistic test datasets tailored to your specific use cases. You provide context (existing examples, domain information, or system prompts), and it creates robust test data that helps you do things like: balance distribution across your test suite including edge cases, evaluate prompt effectiveness across various scenarios, compare and optimize model selection, identify and handle edge cases systematically, assess performance across diverse use cases, support rapid R&D experimentation with different data shapes.

We're launching this as the first data studio focused on giving AI engineers both speed and precision in their development workflow. If you're building AI products, we'd love your feedback on our approach to making LLM testing more reliable and efficient.

Check out what we’re building at https://www.withcoherence.com/