Benchmarking retrieval for agents on messy real-world company knowledge

Benchmarking retrieval for agents on messy real-world company knowledge(kapa.ai)

23 points by emil_sorensen 6 hours ago | 2 comments

emil_sorensen 2 hours ago |

OP/founder of kapa (YCS23) here. Happy to answer any questions :)

polotics 2 hours ago | |

ok I bite!

What is your opinion of:

- BEAM

- MemEval

- MemoryArena

- any other you can find on eg. HuggingFace

How much value do you see in Karpathy's gist on the Episodic/Procedural/Semantic split (aka. btw. Doxa/Koine/Gnosis) If you do see value what parts of your design matches?

As you are using real company data, will you at least make one step towards reproducibility by publishing some extraction pipeline so other companies can run their own comparison?