We've built a data generation studio that helps engineering teams create robust test datasets for their LLM applications in a fraction of the time it usually takes. While working with hundreds of engineering teams on DevOps automation, we discovered a critical gap: everyone wants to build AI products, but acquiring, labelling, and organizing test data for tasks is a massive challenge.

The current approaches to creating "golden datasets" for LLM testing are either infrastructure-heavy (observability-based) or time-consuming (manual creation). Many teams end up relying on intuition-based development, which leaves crucial questions unanswered about model selection, edge cases, and performance optimization.

Our solution generates comprehensive, realistic test datasets tailored to your specific use cases. You provide context (existing examples, domain information, or system prompts), and it creates robust test data that helps you:

- Balance distribution across your test suite including edge cases - Evaluate prompt effectiveness across various scenarios - Compare and optimize model selection - Identify and handle edge cases systematically - Assess performance across diverse use cases - Support rapid R&D experimentation with different data shapes

We're launching this as the first data studio focused on giving AI engineers both speed and precision in their development workflow. If you're building AI products, we'd love your feedback on our approach to making LLM testing more reliable and efficient.

Check out what we’re building at https://www.withcoherence.com/