Trained KV cache bank turns any LLM into Jev like Model

Trained KV cache bank turns any LLM into Jev like Model(gambler-relay-us-west1.leo-fish.ts.net)

5 points by faangguyindia 9 days ago | 2 comments

ramigb 9 days ago |

That's awesome I came from the jev in python thread. is the model shareable? do you have any more writing on this? would love to read more about it! cool demo anyways.

faangguyindia 9 days ago |

Here you can see a 26B model hacked using a trained KV cache bank, responding like Gemma. The latency I am seeing is <127 ms.

ramigb 9 days ago |

That's awesome I came from the jev in python thread. is the model shareable? do you have any more writing on this? would love to read more about it! cool demo anyways.

faangguyindia 9 days ago |

Here you can see a 26B model hacked using a trained KV cache bank, responding like Gemma. The latency I am seeing is <127 ms.