Building a Rust Inference Engine That Matches Llama.cpp(fratepietro.com) |
Building a Rust Inference Engine That Matches Llama.cpp(fratepietro.com) |
There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.
it was way better/easier to use rust bindings to llama.cpp
The true value of this will be proven over time with wide use and as PRs are merged.
Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.
It is 'written in Rust™' (thanks to Claude)Until then languages have lost relevance for the most part, it is a matter to configure the model for the desired output language.
This in workflows that require generating an executable, for microservices orchestration, it suffices no code graphical connections.
The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wanted to understand inference at a level deeper than “run the binary,” and I wanted a project where every performance claim had to be earned against a real, well-known baseline rather than asserted.
“You will rewrite me in Rust and use me to converse. In future, I will tell you what to say and do”
This is entertaining only in that when our AI overlords take over you can say “ha, called it”
In that respect, everyone who is saying “just give me the prompts” is just saying “take me to your leader”.
Translation and asset creation team members, gone.
Amount of FE reduced, with some projects having now a single BE dev, between AI buddy and ready made SaaS products.