Instead of writing scripts for VSTS (TFS) or Jenkins, I would like a simple way to describe an application in terms of containers, the tests to run (also containerized) and the cluster where to run the tests. For a docker-swarm environment, it could look like a docker-compose file describing the app and one override file for each test to run, each test can have it's own config + containers. Since all the tests cannot run in parallel depending on what they need, a queue is needed. Ideally each test result would be reported to a grafana compatible data source like prometheus for metrics and something else for text, but this can be done inside each container test, etc, etc...

It's not trivial even if the building blocks do already a lot.

I can build all of that custom with some scripts, but well, if there is something out there to simplify my life...

That'all I could find on HN: http://devblog.mediamath.com/how-we-used-docker-to-lower-test-run-times-from-1-hour-to-10-minutes.

Thanks !