llama.cpp

140 points by kristianpaul 2 hours ago | 65 comments

hypfer 1 hour ago |

Old news by now, but you might not be aware that llama-server can do multi-model for a while now,

Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)

Then, any api client can just select a model and the system does the right thing.

It's great software. It just works.

You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.

Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.

karimf 1 hour ago |

Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.

ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.

walrus01 1 hour ago | |

At this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.

itake 1 hour ago | |

does your comment depend on the OS? I thought MLX has better performance on MacOS than llama.cpp

quantumleaper 45 minutes ago | | |

The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.

markasoftware 30 minutes ago | |

possibly hitting front page because this website is fairly new? For me, it's certainly the first time I've seen a one-liner curl|bash installer for llama.cpp, which was basically the only reason to use ollama.

imrehg 1 hour ago |

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.

Two examples:

- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen

- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...

I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...

d3Xt3r 1 hour ago | |

So are there any alternatives which do actually work well with ROCm OOTB?

imrehg 1 hour ago | | |

I just switched to Vulkan, and be done with it. :)

As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...

I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV

etdznots 58 minutes ago | |

The first one multiple contributors highlighted the PR as urgent andits had lots of review but it appears to be waiting for another review and/or someone that owns the affected hardware to test that the PR fixes the issue, it wpuld be easy for you to test and report whether or not it does, and the second thing is not related to llama.cpp at all

Yes ideally there would be testing every hardware + software combo but this costs engineering time and $$$ money, and you are running on master branch, no master branch of any software is stable, inherently, if you run into issues, just stick to the old hash where stuff worked, why are you insistent on both being at the bleeding edge and experience 0 breakage!

imrehg 35 minutes ago | | |

I did report my test results on the first one. :)

The second I didn't say it's any of llama.cpp's "fault", but it is _related_ to llama.cpp since it's being shipped in another system, aye?

Can't stick to the old hash either, because older version have different bugs. E.g. on older versions the same Qwen3.6 model reliably fails to call specific tools due to template issues, while just having the newer llama.cpp version has that fixed. So different versions - different bugs, rather than no bugs.

Why the beating you are trying to gimme, mate? :)

pplonski86 1 hour ago |

Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.

chii 42 minutes ago | |

> I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth

what model was it that you were able to run with the rtx 3070?

tosh 2 hours ago |

I was a bit suspicious of the url but it is also listed on llama.cpp github

https://github.com/ggml-org/llama.cpp

bityard 1 hour ago | |

Same, but it looks legit enough to me. Here is the git repo for the site with a link back to llama.app: https://github.com/ggml-org/llama.pages

(I still deeply distrust curlpipes in general though.)

tosh 1 hour ago | | |

ty for digging this up!

walrus01 1 hour ago |

Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).

Git clone llama.cpp and build it, it's not hard.

https://github.com/ggml-org/llama.cpp/blob/master/docs/build...

literally just a few steps for the basics:

git clone https://github.com/ggml-org/llama.cpp

cmake -B build

cmake --build build --config Release

prologic 1 hour ago |

Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.

HelloUsername 1 hour ago | |

"Friends don't let friends use ollama" https://sleepingrobots.com/dreams/stop-using-ollama/

mdp2021 56 minutes ago | |

> Is llama.cpp (and thus llama.app)

llama.app is just an URL (for the "advertisement" webpages of llama.cpp outside GitHub).

> that much better than Ollama

llama.cpp is the real thing, ollama was a fork that remained inferior.

hhh 1 hour ago | |

ollama uses llama.cpp

larodi 34 minutes ago |

This site seems scam for not noting origins of llama.cpp and fails to quickly and clearly communicate it NOT being affiliated with GGML org.

gr_norm 32 minutes ago | |

From https://github.com/ggml-org/llama.cpp:

> Visit https://llama.app and follow the instructions

It's linked at the start of the README.

blahblaher 8 minutes ago |

and? whats the point of this? Doesn't everyone already know about llama.cpp?

nexawave-ai 1 hour ago |

I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?

dlcarrier 1 hour ago |

I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.

cptskippy 1 minute ago | |

Does the A770 use the Xe driver? If so then it might work with the scripts that I've been using to build llama.cpp with SYCL support for the Arc Pro B70.

https://github.com/cptskippy/battlemage-llm-gateway

It's designed so that you can re-run the scripts to pull the latest updates. When Muse Glimmer was released the other day I just ran the 02 script to build the latest version of llama.cpp with support for it.

madushan1000 1 hour ago | |

Two days sounds like a lot, both llama.cpp and openvino only takes a few minutes to compile on any decent modern cpu.

walrus01 1 hour ago | | |

the full set of llama.cpp binaries builds in under 5 minutes with an unmodified build workflow straight from their github page on a literally ten year old dual xeon.

gnull 1 hour ago | | |

It also even easier to get working and integrate into your system in a sustainable manner with NixOS. Do it by hand or throw an LLM at it, it will get you a declarative patch for your NixOS config that brings llama-cpp into your config that you can review and add under version control (no random `make install` build artifacts contaminating your system, no wondering "what was it that I ran? what are all these files? how do I do the same with a newer version?" a couple months later). There's also likely some build cache where Nixoids have already build what you want.

I had a great experience with llama-cpp with Nvidia backend on NixOS.

(Sorry for being that guy.)

numpad0 34 minutes ago | |

I don't know how it works, but does Vulkan not work on Arc?

halyconWays 40 minutes ago |

llama.cpp is like the ffmepg of AI, and one of the reasons I so greatly dislike ollama is that the latter completely obfuscates that they're a rebrand of the former. Georgi Gerganov and team did all the hard work; ollama is langchain-like VC-bait with a HF download wrapper.

antonvs 52 minutes ago |

> No telemetry

Must be tough not to be able to monitor your own models!

(The odds that that tagline was AI-generated seem high.)

helsinkiandrew 1 hour ago |

I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not

reverius42 1 hour ago | |

It's from https://github.com/ggml-org/llama.cpp -- not associated with Meta, it's been around for years, and surely they know about it -- so I would guess either it's not a trademark violation or they don't care.

bhouston 1 hour ago |

It seems that llama.app is a direct competitor to ollama.com

I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.

fmajid 1 hour ago | |

And ollama were sketchy about not providing proper credit to llama.cpp, even though that’s all they are, a wrapper for it.

ryan_glass 1 hour ago | |

Ollama uses the llama.cpp backend for inference. I find Ollama noticably slower. Llama.cpp has had a built-in webui (used as llama-server) for a long time now so have owned the user experience too.