Q8 KV cache lets a 30B model fit 100K context on a 24 GB RTX 5090 | Dark Hacker News

Dark Hacker News

new|best|ask|show|jobs

Built with the HackerNews API. Developed by LJT.AI

Note: The HackerNews API provides up to 500 stories per category

Q8 KV cache lets a 30B model fit 100K context on a 24 GB RTX 5090 | Dark Hacker News

Q8 KV cache lets a 30B model fit 100K context on a 24 GB RTX 5090(buraak.com)

3 points by bozdemir 14 days ago | 0 comments

No comments yet