Train Your Own LLM from Scratch(github.com) |
Train Your Own LLM from Scratch(github.com) |
A series of Jupyter notebooks explaining the whole machine learning mechanism, from the beginning
https://github.com/nickyreinert/DeepLearning-with-PyTorch-fr...
and of course also how to build an llm from scratch
https://github.com/nickyreinert/basic-llm-with-pytorch/blob/...
The engineering was horrible and very ad-hoc but I learned a lot. Results were ok-ish (I classified tweets) but it gave me a good perspective on the sheer GPU power (and engineering challenges) one would need to do this seriously. I didn't fully grasp the potential of generating output but spent quite some time chuckling at generated tweets (was just curious to try it).
[0] https://github.com/rasbt/LLMs-from-scratch
[1] https://www.manning.com/books/build-a-large-language-model-f...
[2] https://magazine.sebastianraschka.com/p/coding-llms-from-the...
As particular motivation for my intuition, I expect that we had evolutionary pressure to adapt our defense mechanisms of predicting the movements of predators and prey, to handle human opponents.
But that is just me. I think is more useful to understand the how and whys before training a LLM.
...nanoGPT targets reproducing GPT-2 (124M params) and covers a lot of ground. This project strips it down to the essentials and scales it to a ~10M param model that trains on a laptop in under an hour...
I see in dependencies torch, so most likely tensors and backpropagation are not implemented, but rather taken as granted. Does it count then as writing "from scratch"?..
I did something similar (in Rust, AI assisted), but I restricted myself not to use any dependency, only standard library. As result, I have to implement much more things, such as tensor design, kernels concept, simple gradient descent optimizer and even custom json parser, cpu data parallelism abstractions similar to rayon, etc. It was quite fun when I got everything wired and working - soo sloooow, but working.
I doubt you have a machine big enough to make it "Large".
I'm not saying it's worth it but you don't need to buy a GPU yourself to be able to train.
And it's paired with 48 processor cores! I mean, they don't even support AVX512 but they can do math!
I could totally train a LLM! Or at least my family could... might need my kid to pick up and carry on the project.
But in all seriousness... you either missed the point, are being needlessly pedantic, or are... wrong?
This is about learning concepts, and the rest of this is mostly moot.
On the pedantic or wrong notes--What is the documented cut-off for a "large" language model? Because GPT-2 was and is described as a "large" language model. It had 1.5B parameters. You can just about get a consumer GPU capable of training that for about $400 these days.
In my own very humble opinion, it becomes "Large" when it's out of non-specialized hardware. So currently, a model which requires more than 32GB vram is large (as that's roughly where the high-end gaming GPUs cut off).
And btw, there is no way you can train a language model on a CPU, even with ddr5, lest you wait a whole week for a single training cycle. Give it a go! I know I did, it's a magnitude away from being feasible.
runs on a Blackwell 6000 Max-Q, using 86GB VRAM. Training supposedly takes 3h40m
I'm not sure. Microsoft calls Phi-4 a small language model, so the distinction is considered meaningful to some people working in the space. My own view is that the term "LLM" implies something about the capabilities of the model in 2026. Maybe there's not a hard definition of the term, but whatever the definition is, the model in the article wouldn't make it.
GPT would have been a better term than LLM, but unfortunately became too associated with OpenAI. And then, what about non-transformer LLMs? And multimodal LLMs?
Maybe we should just give up, shrug and call it "AI".
Sure, we could do it like we did radio frequencies! Most of what we use are "High Frequency" and above... Very High Frequency, Ultra High Frequency, Super High Frequency, Extremely High Frequency.
> In my own very humble opinion, it becomes "Large" when it's out of non-specialized hardware. So currently, a model which requires more than 32GB vram is large (as that's roughly where the high-end gaming GPUs cut off).
So the definition shifts over time based on the market availability of RAM? And can also go backwards? I can't really see anyone bothering to look up the state of the GPU market in order to determine correct terminology whenever they want to talk about this stuff (or interpret old comments, or...).
That also decouples the terminology from the actual capabilities which is what people are generally more interested in. GPT-3 was a "large" language model at this present time. However the the seemingly much more capable Gemma 4 was a large language model at the time GPT-3 was in use, but isn't a large language model right now.
I kinda question the arbitrary line drawn here too--32GB VRAM? Where I am that's a ~$5-6k problem. I'm not sure I'd call that a "consumer" product any more than the $20k data center cards regardless of the OEM intent, but we could argue semantics on that one too.
Fundamentally, defining it this way just seems kind of... useless? It's borderline a meaningless modifier already. This just defines it in a way that's so complex to use or interpret that it's just meaningless in a different way.
For what it's worth, I'd vote to use "large" to mean "big enough to be general purpose", more differentiating from the small, specialized models that came before.
> And btw, there is no way you can train a language model on a CPU, even with ddr5, lest you wait a whole week for a single training cycle. Give it a go! I know I did, it's a magnitude away from being feasible.
Yeah, was mostly being silly--tried to allude to that with the "intergenerational project" comment toward the end there.
Though I _did_ try doing some inference on CPU, which is how I found out that these Xeons I have don't implement AVX512. Surprisingly Gemma 4 (2B) was able to spit out a solid 13-14 tok/s! Was expecting more like... 0.13.
And no one is stopping anyone from tweaking few parameters in this repo to go above 10M parameters.
If you want to be snarky, it helps if you are right.
He could have done that initially instead of saying "Google the name of the author."