We've been doing training (Dreambooth) and inference on TPUs since the beginning of the year at https://dreamlook.ai.
We basically get 2.5x the training speed for Stable Diffusion 1.5 compared to A100, a very nice "unfair advantage"!
I just tried it on my RTX 3090, with a riced linux environment + pytorch/xformers nightly, and 4 images take 36.7 seconds on the ComfyUI backend (used by Fooocus-MRE).
...But the issue is, right now, you can either pick high quality tooling (The ComfyUI/automatic1111 backend UIs) or speed (diffusers-based UIs), not both. InvokeAI and VoltaML do not support SDXL as well as Fooocus at the moment, and all the other UIs use the slow Stability backend with no compilation support.
See https://github.com/invoke-ai/InvokeAI/releases
I believe invoke supports caching and a number of interesting features this release.
I really wanted to use Invoke for the better performance and torch.compile support, but the switch to the refiner seems slow, and torch.compile doesn't seem to work either. I need to investigate torch.compile more, but maybe they did something to the vanilla diffusers pipeline?
When used with ControlNet it still runs but more layers offloaded at 1.4s/it