Why is Chat GPT so expensive to operate? Altman has said "it's a few cents per chat", which probably means it closer to high single digit cents per chat. Does that estimate include amortization of upfront development costs, or is it actually the marginal cost of a chat? |
Meta released their OPT model which they claim is comparable to the GPT-3 model. Guidance for running that model [1] suggests a LOT of memory - at least 350GB of gpu memory which is roughly 4 A1000s, which are pricy.
Running this on AWS with the above suggestion would cost $25/hr - just for one model running. That’s almost $0.50 a minute. If you imagine it takes a few seconds to run the model for one request… easily you’ll hit $0.05 per request once you factor in the rest of the infra (storage, CDN, etc) and the engineering cost, and the research cost, and the fact that they probably have a scale to hundreds of instances for heavy traffic and that may mean less efficient purchased servers.
OpenAI has a sweetheart deal with Azure, but this is roughly the cost structure for serving requests. And this doesn’t include the upfront cost of training.
Yet people these days believe something like the brain was bruteforced by nature into an accidental existence.
Stable diffusion can run on a home pc, while it seems you need a super computer for GPT3. I'm not sure that would have been my intuition.
- run tokenization of inputs on CPU
- sort inputs by length
- batch inputs of similar length and apply padding to make of uniform length
- pass the batches through so a single model can process many inputs in parallel.
For GPT-style decoder models however, this becomes much more challenging because inference requires a forward pass for every token generated. (Stopping criteria also may differ but that’s another tangent).
Every generated token performs attention on every previous token, both the context (or “prompt”) and the previously generated tokens (important for self consistency). this is a quadratic operation in the vanilla case.
Model sizes are large , often spanning multiple machines, and the information for later layers depends on previous ones, meaning inference has to be pipelined.
The naive approach would be to have a single transaction processed exclusively by a single instance of the model. this is expensive! even if each model can be crammed into a single A100 , if you want to run something like Codex or ChatGPT for millions of users with low latency inference, you’d have to have thousands of GPUs preloaded with models, and each transaction would take a highly variable amount of time.
If a model spans multiple machines, you’d achieve a max of 1/n% utilization because each shard has to remain loaded while the others process, and then if you want to do pipeline parallelism like in pipe dream, you’d have to deal with attention caches since you don’t want to have to recompute every previous state each time
The answer is almost certainly "no." A service like Chat GPT is expensive because it requires heavy-duty GPU computations.
I don't think these numbers sounds very out of line. It would be easier to understand the feasibility of this if we knew how fast those cards could execute the model. If it takes a second to run it then a few cents seems about right, if it takes a few milliseconds then it is a lot less than a few cents unless Microsoft charges huge premium for the servers.
Now there are different ways to achieve this, but in essence because it has to know everything all at once plus instructions on how to handle that.
You can actually ask it to explain to you how you could create a natural language processing algorithm yourself and it will even give you a starter framework in the language of your choice. But a fair warning, for me it was like a 6 hour deep rabbit hole :D
Read the papers.
Is the H100 deployed at Azure? I wonder how much more efficient that would be over A100s.
Memory shipped with computers have been stagnate for a decade
We live in a really exciting age :). Local AI models will also finally give Microsoft reasons again to require hardware for coming Windows versions. Now they have to require obscure security chips and stuff but in the future they might have some local cortana thingy or something that requires a certain amount of computational power.
It will write 200 lines of code for me which would maybe take me a few hours. I have to spend 15 minutes cleaning it up, but still it saved me 80% of the time. It's a massive win.
Also great for writing articles, or emails. I write what I want to say into ChatGPT and tell it to state rewrite it to be pleasant and less harsh and it does a great job of that.
For example, it gives you code. You run that code to see if the outputs are as expected.
This hubris over the top-level language in the system is so passe, so 2000s.
https://www.robnugen.com/journal/2023/01/14/chatgpt-helped-m...
Outcome should be: The organism successfully produces offspring
Natural selection is doing exactly what you describe.
There doesn't have to be a high rate of survival if the reproductive rate compensates for losses.
E.g., if 80% of wild rabbits are eaten, but the remaining 20% can give birth to 5 bunnies per parent per lifetime, the population will be stable.
I have no idea where you're getting your beliefs, but most of it is wrong in both the math and biology.
if we assume that a computer can be powered by 100 watts, over a day it will use 2.4 kW h, which is about 2000 Calories
GPU will consume a lot more, but we aren’t that far off in efficiency
Look at it in bits and bytes. For each adversity overcoming feature that a species has inherited, let that a be a bit set to 1. With 2 adversaries you have only two bits where only need one out of 4 individuals that has both bits on. For a realistic adversity of 32, you need 4billion bits all set to one. And this is without considering how a survival trait against one adversity can be a fatal trait against another. Now these bits need to be passed on, if one of them is missing then the only chance that individual has to survive is by pure chance they avoid that adversary.
Think of the endless adversities we face and overcome, you are saying for millions of generations, there has been an unbroken chain of survivors that kept overcoming a geometrically expanding adversity. Just a degree increasing in the global temperature causes entire ecosystems to collapse.
Survival is the exception, not the default.
I think you’re ignoring a bunch of dynamics by trying to model it with binary.
Prey population going down means a predator population also going down and a competitor population going up. It’s not an endgame, it’s just an “ear” of a very complex attractor, which with time only sharpens it ability to have as little escape points as possible.
1°C fluctuation by itself does nothing, because life usually has much wider tolerance due to long seasonal fluctuations. A global +-degree means there will be a tipping point somewhere which would bring a local drastic change. Locally life may suffer, but it counteracts with migration and preexisting diversity. It simply suffers everywhere, always. It’s a modus operandi. A little bit more is barely fatal.
So yes, survival is the default because life naturally specializes in it.