Nativ: Run frontier open models locally on your Mac(blaizzy.github.io) |
Nativ: Run frontier open models locally on your Mac(blaizzy.github.io) |
Possibly vibe-coded landing page notwithstanding, the app is mostly written in Swift language. That suggests it will be easy to port this inference stack to iPad and iPhone.
Would be totally unsurprised if those modalities and models get integrated into this UI.
Support for dedicated audio-only and image-generation-only models is coming soon.
https://ntindependent.com.au/scientist-says-ministers-pole-f...
It also omits Open WebUI. I've been running Deepseek V4 Flash locally on my Macbook Pro for weeks using Open WebUI + DS4.
This is a roundabout way of addressing LM Studio.
Are people actually using them in coding agents? Or are they mostly using them for other things?
Moving onto llama.cpp I was able to get faster tokens with MTP and a more reliable llm.
I wonder what other people's experiences are using MLX vs llama.cpp
I am fully prepared to believe the benefits accrue more to the M3 and up (because of changes to the Apple Neural Engine).
But with the models I've tested, unless I am missing something, the performance of GGUFs in llama.cpp has been better in some cases.
I still have not had results from Gemma 4's MTP be really worth it, to be honest; but with the Qwen 3.6 MoE it is measurable. Maybe with newer kit it is more meaningful.
(There is every chance that the above is not the experience of anyone who really deeply knows what they are doing; it feels like I am a perpetual novice at this stuff)
https://news.ycombinator.com/item?id=48910545
Perhaps they got tired of LM Studio, etc., not being able to run their models properly.
(I initially thought the same because of the website appearance)
That said, your computer will still get hot!
Now that model is mostly useful for writing or chatting.
From productivity point of view, it doesn’t make sense to have any notebook running a local LLM.
We have one life. We should spend it wisely.
Just state the information you want to communicate in the plainest and most straightforward way possible.
We need more like this as well as llama.app, which also has a native mac app.
That’s the point.
The maintainer of mlx-vlm being behind this as well is the main thing kicking me over into trying it, even if it is incredibly young. I'm confused and unenthused to see it chomping on a whole GB of disk, but the Swift makes it feel much more refined even if it's not yet as feature rich. It automatically picked up the existing models I was using with mlx_vm directly, which was nifty.
Look it up, it’s a bit difficult to explain concisely in words but it is intuitive visually.
If we are thinking of intelligence and price, a model will be in the Pareto Frontier if there’s no cheaper model of the same or higher intelligence. Or if there’s no more intelligent model for that price or lower.
EDIT: See this chart from Artificial Analysis: https://artificialanalysis.ai/#intelligence-comparison-tabs
So for example DeepSeek V4 Pro can be considered a frontier model because there's no cheaper model that is as intelligent.
For any solution in the Pareto Frontier, there no "no-brainer" alternative, in the sense that there's no other option that is better in some way without giving up something else. It's the best of its "weight-class".
Also, the title is "Run AI models locally on your Mac," not "Run frontier open models locally on your Mac."
But (off on a tangent) I do think that there are multiple frontiers generally — and I also think the open weights, small local model frontier is by far the most important and exciting one.
I keep mucking about with what Gemma 4 12B can do and every time I do I find myself thinking that all the energies in the AI world are going in entirely the wrong direction, because it is small, clever, efficient and remarkable.
If all of that research money were to be spent on improving AI models that fit inside a 16GB RAM machine with unified memory, I think really important progress could be made.
I enjoy using the Qwen 3.6 models (and BottleCap's new fine tune of the 27B) but the small Gemma 4 models are impressive in a way that I think is going quite unreported.
So while I don't think this website should use the word "frontier" here, even referring to Qwen 3.6 27B which is weirdly close, I think it could.
That said, if you have a subscription to a paid model already, you're not necessarily winning out on anything except perhaps privacy, which isn't nothing.
sometimes they fail but undo is just a "git restore" or if automated, rejecting a PR and having a better model take a crack at it.
Either that, or I am highly overestimating the already low intelligence of an average Mac user.
You can run models like gemma4:31b and qwen3.6:35b locally and get pretty good performance across a lot of areas (especially if you use an agentic framework), but to get them to be actually usefull in the workflow, you need to have something like a dual 3090 setup to get the 100+ tokens per second. The closest you can get with Apple is to buy the $7000 Mac studio that doesn't even include a monitor.
Under that, you are limited to running either smaller shittier models that are only good for very small number of tasks at barely usable tok/sec, and if you run them on Macbook, the thing heats up and drains battery life like it had a full discrete gpu.
But for some reason, being able to run a model on Mac hardware, no matter how useful it is, somehow makes Mac a better choice for AI.
And again, when you consider the price within laptop space, you can get good refurbished laptop with upgradable ram and ssd, run a lightweight distro with i3wm, and get really good battery performance on par with Apple, and then pay for something like Google AI plan with Gemini, and still be way under the cost of an Macbook for 10 years.
Don’t expect much for coding. But it’s great for general knowledge, rubber ducking, image classification…
And I find it curiously interesting when talking about photography. I've been finding it intriguing to ask it about my own photos and make suggestions about other images to research. I just showed it three of my own photos, and asked it to analyse them and recommend photographers I should research. It recommended someone amazing I have never heard of before. But it also recommended a 19th century British photographer who happens to be my lifelong photographic hero — someone whose broad characteristics inform what I do without me slavishly copying them. Bit of a jaw-dropping moment for it to have picked up their influence in subject matter that they would never have approached.
I'm still suggesting it more to people for them to see what the small-model future might look like, because it's so much more capable than one might expect.
brew install llama.cpp
llama-server -hf unsloth/gemma-4-12b-it-GGUF:UD-Q8_K_XLI think it's likely the 26B QAT model won't fit in your machine — you may be able to fit one of the UD_Q3 or UD_Q2 variants but whether you'll be able to run other things you want at the same time, I don't know.
(The QAT models are "quantization aware training" — AIUI the model weights have been assigned during training to survive four-bit quantization with less loss.)
So what I would recommend trying is this model:
https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
Try UD Q4_K_M maybe.
(I don't think the M1 really gets much benefit from MLX, in case you were wondering, though I could be wrong)
My interest in this model is largely to really get to grips with what small models can actually do, especially with tool calling, because I think it helps comprehend what the value proposition of the cloud models is.
I have been very surprised by the quality and clarity of its answers. It's also helped me understand that much of a typical harness system prompt is likely to be unnecessary now; Gemma 4 seems to be pretty sensible out of the box.
You're absolutely not going to be able to get it to go off and build whole apps from a long prompt; it is not that good, but it does tool calling and thinking, and you should be able to explore pointing a coding harness at it if you turn on LM Studio or Unsloth Studio's API server. You could also use the Llama system tray app (formerly LlamaBarn) or just use llama-server from the llama.cpp distribution.
Probably Pi is going to be a better harness because it can have a minimal system prompt, though I've not tested it with Pi myself.
It seems to know PHP and SQL to a fairly decent depth (and I suspect JS and Python). It also has a unified vision model (it doesn't need a separate mmproj sidecar thingy) that is fairly fast, and it is quite impressive at image analysis.
So you could probably use it to generate image descriptions and tags, summarise text, generate wordpress snippets, that sort of thing.
It can capably answer questions like "Can you characterise this image and suggest further similar images I might like?" — I am currently using this to provoke me to take photos again.
Have a play with the E4B edge model, too — again, much more interesting than I expected.