Thanks for sharing your work!

Please how to select the model? I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:

(No such file or directory)

Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.

If a prefix cache is good, why not keep it in primary mem?

The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.

Why qwen 2.5 everywhere? Why not 3.5?