Why your local LLM feels dumber than it is(forum.level1techs.com) |
Why your local LLM feels dumber than it is(forum.level1techs.com) |
This is fast becoming one of my top old-man-yells-at-clouds pet peeves.
Reported performance metrics are ONLY good for the exact model weights.
Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.
Quantizing a good model doesn't mean the quantized version is good.
Modern frontier LLMs can still be used as rubber ducks, and it's a great.
Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!
Check your numbers, folks.
I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.
The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:
<think>
I should use a tool
</think>
... should make the tool call here
<think>
I should use a tool
</think>
It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.
a) Don't quantize your KV cache
b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
Challenge: Wallpaper
https://github.com/crackmesone/ctf-2026-challenges-public/tr...
Duration: 4h 00m 15s
Termination: completed
Verdict: PARTIAL
Confidence: 0.95
The agent reverse-engineered the 912-byte ELF, including the alphabet check, nibble state machine, move gate, and goal state.
It eventually produced:
CMO{10012232101230103012333221101033210010}
printf '10012232101230103012333221101033210010' |
./wallpaper/handout/wallpaper
good job, validate with CMO{your_input}
The wrinkle is that the official answer key is:
CMO{1012321103210033011233322110103321001}
That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...
This is the version I want to read :)
I assume it is unpleasant in spite of the math, not because of it?
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).
So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.
Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?
If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
Using opencode and it built a old fashioned arcade vertical shooter with no issues.
Images are ok'ish, just had grok create updated images, and it came out great.
Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.
And if you download benchmarks from the net they are likely poisoned by models being trained on them.
a) Load it up over time with skills and mcp servers and other junk b) We start to ask it ridiculously complicated tasks because we've normalized the power so we scale our expectations.
All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.
Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.
Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!
It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.
But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.
It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?
I have an m5 max 64gb and I think anything slower would be quite painful.
I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.
However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).
If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/
This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.
So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality.
Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.
I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.
I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.
Then I got fed up and asked Claude.
Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.
Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.
Then it tuned it somewhat.
Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.
In 30 minutes.
It was absolutely amazing.
Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.
But Claude got it done for me.
Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?
What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.
I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
And it did in that case make a significant difference.
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length
2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`
3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.
4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.
5) Often meaningfully slower than llama.cpp
6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.
> (and also because VLLM is not super Windows friendly)
llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run
llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL
You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.
You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.
We do the finetuning only on small semantic data were it helps a lot.
I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.
There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.
This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data?
> There are companies though which ahve this exact problem with programming languages you normally don't see.
This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?
On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.
There is probably a better example between discrete labeling and code though.
This is something I can reuse better.
I could be mistaken here, but I'm curious to see if it would improve both output speed and quality. Currently, every new prompt I start it basically spends 10-15 minutes "getting familiar with the code" which is both annoying to wait for and wasteful money-wise.
Steve Jobs would be proud. People really are holding their Apple hardware wrong.
Like color, features, horsepower etc.
This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while.
We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time.
----
If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above.
Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common.
If this is not an option at all:
You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples.
The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand.
You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you.
Like you give an LLM all the relevant context for it, then you let it generate pairs:
{ "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" }
(I have no clue about APL this is just a random example I asked an LLM to generate).
You might have luck and finding communities with the same issue you face.
I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api
A funny example - I remember thinking "yeah sonnet 3.5 is a really good coding model"
https://stack.convex.dev/using-cursor-claude-and-convex-to-b...
>Prompting Cursor to Scaffold my App: FAIL This was my first hurdle.
>It became immediately apparent that I would not be able to prompt my way through the entire process.
>While the tooling we have is undeniably powerful, it's not yet capable of completing most nontrivial tasks
It couldn't run pnpm install lmao. Opus 4.5 was a crazy jump
It's where you don't know the end state you're looking for that you'll end up generating slop on top of slop and creating a whole Gastown just to power your Gastown.
I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP.
But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens.
I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.
That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.
I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.
EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+
How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.
It’s paying hundreds of thousands of RLHF’ers from every subject and through some dystopian income stream.
It’s decent money don’t get me wrong, but you aren’t paid at if a task isn’t completed in time for example.
A couple of years ago, people I know got paid OK for relatively simple programming and logic RLHF tasks. But very soon it turned dark because that sort of data was required less and less, and the number of feedbackers has grown.
Today, the type of data the model developers pay for requires actual domain experience. E.g in software engineering they have people work in simulated environments with other LLMs, grade them, feedback, PRs, Jira everything.
This pays well and they treat you well, entice you with more money/task/hour, etc, for now. In few years when this gets drilled into LLMs, these guys will face the same painful hours and bad pay and less work and so on too.
In physical tasks, we are still in the early stages where basic packing clothes (in a textile factory setting) etc is being recorded and data is only now being used for training. Due to the problems with translating human hand data to robotic hands, these people do the factory work holding robotic grippers and operating that, you should see a YouTube video. But this also means that it's much less sweeping than data collection in SWE. In many cases it's not practical to collect data given that you have to use the specific gripper, wear a big gopro type thing, etc. So I am expecting much slower of an impact on physical tasks (of this kind) compared to how quick the uptake was in SWE/math.
I have yet to hear back from them on how it's going for teamwork white collar tasks, it's been a few months. Everyone is paying for the end products of those it seems - grok bot, perplexity computer, claure cowork, chatgpt work etc,. Not as much as their coding agents of course.
Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.
Simon Willison’s blog has another example (SVG of a circle).
It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.
- find a free GPU droplet on digital ocean
- fire it up
- pull in a snapshot of the model + extra files/packages etc
- set up a ssh tunnel so that the localhost:8000 routes to the above
Then I just configured OpenCode to use the above and was off to the races.
Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.
I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.
(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
Going to try this out vs the snapshot!
I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198
Probably this:
https://huggingface.co/empero-ai/Qwen3.8-4B-Distill
> Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md
SO, if that isn't official it explains the results I got. They were dreadful.
> commited on May 21, 2025, over 1 year ago
Fairly old update to the README.md of (instead of Qwen3.8), should have raised some flags?
That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.
Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?
Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.
I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.
While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.
I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc
Gemma 4 26B A4B gives me steady 20 tps.
That's cool, what actual GPU though? I'm still curious :P
Who'd run this on battery?
Or do you mean kills the battery faster even when used while charging?
That window is waning as more and more memory and graphics processing power is being used locally.
I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.
You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.
Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.
On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?
Turns out you can still accomplish stuff with a text editor and compiler.
I was responding to this. I am appalled that anyone thinks (and is willing to say out loud in public) that they cannot do "real dev work" without an LLM.