I was wondering what is HN's take on the state of the art when it comes to hosting GPU-based Machine Learning model demos. We are a university unit with a strong focus on Applied ML and the need to host research applications and student project applications that rely in GPU-based models.

We currently focus on the automation of the CI/CD of these demo applications and prototypes that are based on computer vision models or LLMS. We are aware of solutions that cover the pipeline aspect of CI/CD such as Gitlab, Github and the ML OPS platforms.

Where it currently breaks down for us, is the identification of hosting that would allow us to deploy a range of demo or prototype applications.

Our requirements are:

- host a number of applications / demos at the same time

- only prediction is of interest here (training happens on other machines)

- in many cases there is the need to host our own trained models

- GPU in the range of 12-24 GB GPU

- traffic would be minimal

- minimum redundancy or fault-tolerance levels

- would like to keep expense at level of 500-1000$ per month

The hosting platforms we are currently evaluating are:

- https://www.centron.de/en/centron-cloud-gpu-services/ - https://www.ovhcloud.com/en/public-cloud/prices/

To use these platforms to host multiple demos would it be the easiest to run Kubernetes and deploy Pods? Are there good best practices to host multiple low traffic web applications based on this setup? Or maybe there exists a much simpler setup to share a GPU between applications.

Thanks a lot in advance for your input. This will directly flow into the hosting implementation but will also find its way into the teaching after review.