I’m currently trying to solve a problem we're having, GPUs are expensive! I've been thinking of ways to cut our inference costs at my company and wanted to hear your perspective.

Did anyone implement something similar? How did it go? How much time did it save? What was the cost improvement? I recently found this tool in the AWS samples: https://github.com/aws-samples/scalable-hw-agnostic-inference

I'm wondering if anyone used/tried it or other approaches?