Popping the GPU Bubble(moondream.ai) |
Popping the GPU Bubble(moondream.ai) |
To work in LLM training/inference you’re expected to know this stuff but to know this stuff you need to be working in the space.
I guess the difference here being that we have ample compiler literature and practically know 99% of all there is to know about compilers that exist in the wild vs this new field.
Until we’ve gathered and agreed on a few “dragon books” for LLMs and have explored all there is to LLMs, you’re probably right - know-how will be with the practitioners and in source code until it’s distilled (pun intended).
First, where do you know exactly what the optimal VRAM assignment per model, per context size is, which seems to be currently based purely on experience and second how do you make sure that only that amount is available to your infra/containers, which is being handled by DRA and stuff like https://project-hami.io
While only tangentially related to the blog post here. The title is picked in such a way that I couldn't help, but put the shameless plug here. When he wrote popping the bubble, I thought we're talking about devices and reducing NVIDIA dependency, but this seems very focused on Cuda.
Disclaimer: I work with Dynamia.ai, the founders of which created HAMi.
Can you explain what you mean here? Are you talking about small neural networks doing specific tasks?
This appears to be different than the recent "Speculative Pipeline Decoding" paper: https://arxiv.org/abs/2605.30852
In general, for some reason CODEX loves CUDA-streams, it's the first optimization it goes for every time when writing GPU kernels. However in many cases this is not a bottleneck, it happens to be so here because the model in the blog is small (2.4ms FW-pass is tiny, and 9B params sit on a single GPU). Large models are closer to 30-40ms. The CPU-GPU sync is 1-2ms, when working on larger MoE models the scheduling of tokens in this way is much less important than for example scheduling of computation/communication or kernel optimization.
I wish the blog would state this at the start with the premise of what has been done, or show that this is indeed the bottleneck with some benchmarking. Otherwise is kind of overselling things imo.
If you scroll down to the section titled "A cost model for the bubble", you will find both benchmark results and us saying, "you get back anywhere from a few percent to a third; more the faster your accelerator/model is".
The benchmarks you point to in the section titled "A cost model for the bubble" dont include any CPU overheads or the T_block-T_pipe you mention, they just give the improvement %.
In general, you answers here in the thread read as defensive and unhumble. They leave a sour taste of your company, you should consider how you engage with your audience.
This is true, but I've never heard anyone refer to this as a GPU bubble before.
I think most people hear "GPU bubble" and think of a financial bubble of some kind.
Even if the AI companies decide to use their own ASICs, they will rather slowly, but continuously introduce them, while removing GPUs that have reached their end of life.
The GPU would be the propeller, the influx is the work, and the operational parameters is what this article's about.
Very odd, but perhaps more familiar to graphics programmers? I will say I'd probably call it a stall, which is exactly what the Vulkan docs call it moments later, so :shrug:
It's actually a very common technique in rendering to not always be able to easily fill in the gaps, that we frequently deliberately introduce an extra frame of latency, so that the GPU is rendering jobs for the latter half of rendering passes for frame N+1 and the early half of rendering passes of frame N+2 while frame N is visible. This still means that a frame takes the same total GPU time to render, but means that the gaps between jobs on a single frame can be usefully filled with something else from the other.
any time your GPU is idle = you are losing $$$ = your TCO is going up. you don't want that.
Better term, anyone?
Maybe AI is a bit of a misnomer, since everything ML at some point just started getting called AI.
And don't take it too literally, VHDL could be replaced by other hardware design languages, maybe even at lower abstraction levels.
And a GPU is already pretty optimized for inference, no? Like isn't it a bunch of FP mults? I don't think HDLs do well with that, either.
If such AI models will deliver on their qualitative promises, and just the huge cost is the burden to overcome, custom ASIC might be a part of the solution.
If, on the other hand, AI models will still be unsuitable for many applications because of their qualitative issues, it is a much harder and different problem to solve - in this case, the AI bubble will plausibly burst.