Ask HN: Hardware for learning LLM fine-tuning What would be your suggestions on hardware to either rent or buy to get into LLM fine-tuning without going broke? Got a personal project and would like to start exploring fine-tuning (Llama2?) in the near future. |
Look into qLoRA or 8bit quantization. You won't need a lot of memory if you do it locally with a small model and don't merge the weights. You can also use Modal Labs, RunPod, or Replicate.com. They have Serverless and per second billing which is great for testing inference.
Amazon Sagemaker training jobs can also work. Just be careful not to leave an inference endpoint on if you use Amazon for that too. You can easily go broke.
My intention is to work on news clustering and summarization. So far just by using some "clever" prompts I have been able to generate some pretty good news summaries and I have not started clustering yet. But, I have used GPT 4 so far and my educated guess is that soon enough I will hit some quality / cost limits. So, fine tuning a Llama 2 model with (hopefully) small datasets to improve costs and quality on my specific tasks seems like a reasonable path forward.
Does that make sense? Thank you for your answer!!!