List me AI models that I can run with my hw?

2 points by andrea81 2 years ago | 4 comments

Tired of seeing AI model names with 300m of parameters.

Actually I want to see the ones compatible to run locally in my current laptop (I'm NOT going to spend money to run a model).

Is there a website where I can enter my laptop specs and list me AI models?

declaredapple 2 years ago |

Take # parameters and divide by 2 and then add like 1 to estimate it for 4bit quantization. A 13B model will need ~7GB of memory at 4bit.

Most quantization models start falling apart rapidly after 4bit, but you can go lower...

If you can fit it in GPU memory it'll likely be usable, if you can't fit it in memory be prepared for single digit t/s or less, the only exception is for people with M1/M2/M3 cpus.

PaulHoule 2 years ago |

Use sbert.net if you believe Steven Covey and your goal is for your model to “first understand”.