Ollama Doesn't Know Its GPU Is on Another Machine

Ollama Doesn't Know Its GPU Is on Another Machine(loopholelabs.io)

5 points by shivanshvij 31 days ago | 1 comment

We started an Ollama container on a MacBook. There's no NVIDIA GPU, no CUDA toolkit, and macOS doesn't even have CUDA drivers. Ollama found an NVIDIA GPU anyway: A 128 GB Blackwell GPU on a DGX Spark across the network.

Our anpproach enables this by intercepting CUDA calls and forwarding them to a remote server. It takes one command, requires no code changes, and the application has no idea.