Show HN: MyGPT a toy LLM which can be trained on Project Gutenberg and dad jokes

Show HN: MyGPT a toy LLM which can be trained on Project Gutenberg and dad jokes(github.com)

25 points by disconnection 2 years ago | 4 comments

My puny version of ChatGPT.

This was based on the excellent LLM lecture series by Andrej Karpathy: https://www.youtube.com/watch?v=kCc8FmEb1nY

The main points of differentiation are that my version is token-based (tiktoken) with code to load up multiple text files as a trining set. Plus, it has a minimal server which is a drop-in replacement for the OpenAI REST API.

So you can train the default tiny 15M parameter model, and use that in your projects instead of ChatGPT.

I trained it on 20Mb of Project Gutenberg encyclopaedias, then fine-tuned it on 120 dad jokes, to get a Q: A: prompt format.

This model + training set is so small that the results are basically a joke; it's for entertainment purposes only. The code is also very rough, and the server only has the minimum functionality filled in.

I embodied this model in my talking LLM-driven hexapod robot, and it could give very silly answers to spoken questions.

ferfumarma 2 years ago |

Can we see some examples of the jokes it produces?

disconnection 2 years ago | |

Q: What is a dog? A: To get a frog.

Q: Why did the chicken cross the road? A: Because it was Sunday.

With "Q: " as a prompt, it will make its own jokes:

Q: How do you cross a race with no cold birds? A: Because they did the toothache entirely.

Q: Why did a figureur hit a like? A: Because a joke.

Pure comic genius!

---

The prompt format is:

Q: {user question}

Here is part of the fine-tuning set (real dad jokes from the internet - not what was generated):

Q: What do you call a fake noodle? A: An impasta

Q: How do you organise a space party? A: You planet!

getwiththeprog 2 years ago |

This is a great idea. I want to make a 'pet' for my kid. I can't get them a real dog, so why not a tinyLLM?

Training on guttenberg data is a great idea. What I would do is train it on all the e-books I have that are suitable for kids (I managed to find quite a lot online).

The dad jokes idea is great, please keep doing things along this line.

disconnection 2 years ago | |

Cheers! Yes, I recommend it - it's a lot of fun.

There are probably easier to install and better LLM models than mine for home use. But my project should work if you have enough Python knowledge to install all the libraries. Just throw a bunch of text files into the "data" folder and uncomment "train()" in main.py, and it will save checkpoints of the model parameters into the "models" folder.