I've been thinking about buying a system to run LLMs locally but the price for one that'll run Qwen3.8-27B well is quite offputting to say the least.

What I've been looking at instead is inference providers that use TEE and E2EE to provide cryptographic guarantees that my prompts and responses are only visible to me and the GPU itself.

Despite their docs and assurances of what their guarantees mean, I'm having trouble getting to a point where I'm actually comfortable trusting them with secrets though. Phala for example seems to be E2EE only to the gateway and will then forward prompts to (potentially third party) providers.

Has anyone been down this path and found a provider they feel safe with?

Qwen 3.8 has the same architecture and the same parameter count as Qwen 3.6. Something is not right with the GGUF if it's 2 times slower. The post says "The hybrid attention architecture is new" and says the author's older Llama build from a "couple weeks ago" failed to run Qwen 3.8 because it did not support Qwen35 architecture, but both 3.6 and 3.8 are based on Qwen35 which was released in February 2026. The post doesn't make any sense.

I'm encountering the same behavior. I've tried 4-8bit quants and get 14-17 tok/s with one run that achieved 19. I'm eagerly awaiting dflash2 support in Unsloth or LM Studio, as allegedly that should increase throughput to around 30tok/s, which is the baseline for what I consider at least somewhat interactive.

Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.

Yea Qwen3.8 wasn't fun to use on my 64GB M4 Max either (better than these numbers though), so my new daily driver is Ornith-1.5-35B-A3B-MLX-4bit. I recommend giving that a whirl if you're on similar hardware, it's definitely better than Qwen3.6 35b-a3b which was my go-to before.

https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B

Pretty impressive how the Mac ends up less expensive than the Strix Halo boxes, at least here in Ireland. A 128GB Mac Studio with an M5 Max (the Ultra can only have 96 or 256GB) still costs less than the "GMKtec EVO-X2" or the Nvidia DGX Spark with similar performance. Is it the same in the US?

I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8?

Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.

The cheapest card that will run this model very well is a ln unlocked CMP 170HX. But you can run it on a 3090. I run it on an old spare A6000 Ampere. I think I wouldn’t use anything lower than 60 tok/s though, which you can get with MTP etc. I just use a full vllm stack but some people see a lot of speed with ninfer (there are non 5090 ports).

The large RAM Macs are unusable for inference of dense models as of now. Token generation is too slow.

the fact that this author cannot get qwen3.8-27b run at the same speed as qwen3.6-27b, says the article is not worth reading. the author does not know anything about how to run local AI. 3.8 and 3.6 are the same model with different weight.

both tg and pp speed are so terrible on author's machine.

These numbers are a lot lower than I expected from such pricey hardware.

I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.

I am running Ornith 1.5 Q4 on my work issued laptop which surprisingly has 40gb of RAM and 20GB of “unified” VRAM.

I am getting 50t/s of prefill and 5-6t/s of inference.

You can’t get crazy with it, but it’s pretty nice to just let it work away at simpler prompts overnight or during lunch break or while I am working on other things.

Being able to literally a 30gb blob and then being able to talk with it while running on your machine it’s a different kind of magic compared to a far away magic in the cloud.

Apart from learning, I cant see the point of spending that kind of money and energy to get such awfull token generation speed. Assuming memory bandwith is the bottleneck, is it just a matter of time until we start to see hbm4 based chip able to run qwen3.8 for normal people running at more than 500 token / seconds? Is memory speed the only technological bottlenecks that prevent us from having fast local model?

I am also seeing slower speeds, roughly the same ballpark, sometimes even lower - 10-11 tok/s on M5 Max. If there is that ONE version (GGUF or MLX) that runs roughly as fast as 3.6 used to run, please let me know.

What would be incredible is the 3.8 35B MoE version too, I can run 3.6 with 60 tok/s which is a really, really nice speed.

The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.

This low-effort slop post is misleading. It suggest that 3.8 is half as fast as 3.6, but this is almost definitely because MTP isn't enabled by default. The two models should perform the same - they are for me. When you get AI to think for you, you lose.

Ornith 1.5 35B MoE is very close in most benchmarks, but runs at 100 tk/s where Qwen3.8 27B runs at 30 tk/s (MBP M5 Max 48GB). It's now my go-to model for quick inquiries. Ornith 1.0 had problems with world knowledge, but 1.5 seems very solid on that front. I'll be running some agentic tests today.

Ornith 1.5 35B is so fast that my CPU doesn't even have time to heat up, where Qwen3.8 would turn my laptop into a toaster within a few minutes. I hope agentic tasks will hold up!

Peaks at over 300 tok/s on my 5090 with Dflash2. At maximum context (252000 or so) still get 60 tok/s. And I don't have n-gram optimization working yet, I only have 64G memory.

> Realistic minimum RAM 16GB

When someone says stuff like this what does it mean? A Mac/computer with this much RAM capacity/marker (how would this even be useful in this case?) or 16 or ≈16GB free RAM?

Because RAM usage at least on my Mac is another beast altogether and seems nothing will bring down the RAM usage other tham <rm -rf>ing the / (that is if I can and something survives to notice the miracle)

AI is a killer adblocker ! imagine connecting it to instagram and curating all the images that you actually care about removing all ads !

FWIW, on my Mac Studio I get ~24-27 tok/s generation between 0-16k context in - that's on the Q6_K GGUF with speculative decoding on. I have spent zero effort optimizing/improving this so far but will be trying the 4 bit MLX next (I've tended to find models drop off somewhat below 6 bit but maybe that isn't the case nowadays).

Ollama? Running a q1 quant? I don't think the writer knows what the are doing here to be honest.

You will get better information cruising r/localllama for about 10 minutes.

I'm running the 5 bit quant on a 3060 12G. Get a handful of tokens/s. Prompts take 10-20 minutes to complete but it works.

is it really worth to run urself? the watts drsin and that 100% gpu usage

>I realized I’d been sitting on the one thing most of those threads were missing: a machine that can actually run it properly, and time to measure it.

This was written by AI I take it. The one thing, huh? All those idiots running it on their RTX 6000s at 140tok/s must be feeling pretty stupid for not getting a Mac Studio instead.

Slop; all those LocalLlama threads "you" read had #s too. And Q1 quant? Why? You have the memory...

For me, a HUGE benefit to running local models is that my data stays mine, on my computers only.

Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business.

With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.

For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams).

PS: I wouldn't be surprised if Microsoft runs something similar on their end :)

But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.

The M3 Ultra was a lot cheaper for most of its lifetime, and really the main reason for buying it is if you want a lot of unified RAM, to run bigger models than 27B models.

The new M5 Ultra should deliver ~50% faster token generation (1.2TB/s mem bandwidth); and extrapolating from my M5 Max (since the M5 Ultra is literally just 2x Maxes), probably ~3x faster PP.

But I don't think it's fair to look at this only from monetary ROI vs API. With local models, you get privacy and ownership.

I do not trust _any_ API provider with my most personal information; such as for example, all my messages, emails, daily journals spanning a decade+, all my photos and videos, etc. So it unlocks new use cases that I simply don't feel comfortable with via API.

And a personal assistant with ALL my context and data, locally, has been incredibly useful for me :) Zero outages either, zero "overloaded", etc. Nearly-zero refusals too (I don't run abliterated models; thinking prefill has worked for anything I've wanted to do)

I have 2x 3090s and I get 260 tokens/sec peak (110 avg) with Dflash2 and about 1500t/s prefill with a Q4 quant of Qwen 27b. I didn't buy an overpriced Apple product and it performs much better. It's extremely reliable for Agentic coding and I can run 2-3 simultaneous agents with a full ~260k context window.

I realize the price of NVIDIA has gone up but there are plenty of GPU options from others like AMD and Intel with reasonable performance.

A slow AI tasks that can process my self hosted personal journal, my medical history in fasten, my diet and exercise in Mealie and Sparky Fitness to offer insights once a day or even once a week is better than nothing. Because I would never upload that data to any of the AI companies.