Ask HN: Why Try RAG on the Client Side? Is Local-First RAG Practical?

4 points by MutedEstate45 1 year ago | 2 comments

Came across MeMemo, which runs Retrieval-Augmented Generation (RAG) directly on the client side using WebGPU. No server calls, full privacy, and faster responses—sounds great, but I’m wondering:

What’s the catch? How do you scale vector search or manage embeddings locally? Can this handle complex use cases, or is it mostly for lightweight tasks? How do we navigate browser limitations, UX challenges, or security concerns in a setup like this?

Curious if anyone here has tried client-side RAG or sees a compelling use case for it. Is this approach worth exploring for privacy-focused apps, or are we not there yet?

softwaredoug 1 year ago |

You can do pretty fast (single digit ms) cosine similarity on 1m vectors with numpy on the CPU. And there are small embedded databases like RocksDB with HNSW indices that work well beyond that. Especially staying at like 256 dims. Quantization can shrink even further.

Also locally you’re QPS is very low as you’re the only searcher.

So with enough RAM, and small enough dataset, it should be fine.

MutedEstate45 1 year ago | |

Thanks for your suggestions, especially RocksDB! I’m gonna play around with it today. Do you have any ideas related to use cases? I was thinking something like dynamic RAG for transformers.js