I guess after Kimi K2.5, other vendors are going to the same route?
Can't wait to see how this model performs on computer automation use cases like VITA AI Coworker.
But ultimately, you need to try them yourself on the tasks you care about and just see. My personal experience is that right now, Gemini Pro performs the best at everything I throw at it. I think it's superior to Claude and all of the OSS models by a small margin, even for things like coding.
I was thinking, that maybe it's better to make my own benchmarks with the questions/things I'm interested in, and whenever a new model comes out run those tests with that model using open-router.
URL is down so cannot tell.
Me too!
> I like Gemini Pro's UI over Claude so much
This I don't understand. I mean, I don't see a lot of difference in both UIs. Quite the opposite, apart from some animations, round corners and color gradings, they seem to look very alike, no?
I made it reduce the price of first month to 1.49$ (It could go to 0.99$ and my frugal mind wanted it haha but I just couldn't have it do that lol)
Anyways, afterwards for privacy purposes/( I am a minor so don't have a card), ended up going to g2a to get a 10$ Visa gift card essentially and used it. (I had to pay a 1$ extra but sure)
Installed kimi code on my mac and trying it out. Honestly, I am kind of liking it.
My internal benchmark is creating pomodoro apps in golang web... Gemini 3 pro has nailed it, I just tried the kimi version and it does have some bugs but it feels like it added more features.
Gonna have to try it out for a month.
I mean I just wish it was this cheap for the whole year :< (As I could then move from, say using the completely free models)
Gonna have to try it out more!
1T parameters, 32b active parameters.
License: MIT with the following modification:
Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue, you shall prominently display "Kimi K2.5" on the user interface of such product or service.
Technical awe at this marvel aside that cracks the 50th percentile of HLE, the snarky part of me says there’s only half the danger in giving something away nobody can run at home anyway…
The cheapest way is to stream it from a fast SSD, but it will be quite slow (one token every few seconds).
The next step up is an old server with lots of RAM and many memory channels with maybe a GPU thrown in for faster prompt processing (low two digits tokens/second).
At the high end, there are servers with multiple GPUs with lots of VRAM or multiple chained Macs or Strix Halo mini PCs.
The key enabler here is that the models are MoE (Mixture of Experts), which means that only a small(ish) part of the model is required to compute the next token. In this case, there are 32B active parameters, which is about 16GB at 4 bit per parameter. This only leaves the question of how to get those 16GB to the processor as fast as possible.
Kimi-K2-Thinking-UD-Q3_K_XL-00001-of-00010.gguf Generation - 5,231 tokens 604.63s 8.65 tokens/s
"Good Luck" - Kimi <Taken voice>
Because I feel like they mentioned that agent swarm is available their api and that made me feel as if it wasn't open (weights)*? Please let me know if all are open source or not?
Why not just say "you shall pay us 1 million dollars"?
Coincidence or not, let's just marvel for a second over this amount of magic/technology that's being given away for free... and how liberating and different this is than OpenAI and others that were closed to "protect us all".
> K2.5 Agent Swarm improves performance on complex tasks through parallel, specialized execution [..] leads to an 80% reduction in end-to-end runtime
Not just RL on tool calling, but RL on agent orchestration, neat!
Is this within the model? Or within the IDE/service that runs the model?
Because tool calling is mostly just the agent outputting "call tool X", and the IDE does it and returns the data back to AI's context
* https://lmarena.ai/leaderboard — crowd-sourced head-to-head battles between models using ELO
* https://dashboard.safe.ai/ — CAIS' incredible dashboard
* https://clocks.brianmoore.com/ — a visual comparison of how well models can draw a clock. A new clock is drawn every minute
* https://eqbench.com/ — emotional intelligence benchmarks for LLMs
* https://www.ocrarena.ai/battle — OCR battles, ELO
* https://mafia-arena.com/ — LLMs playing the social deduction game Mafia
* https://openrouter.ai/rankings — marketshare based on OpenRouter
That's cool. It also has a zsh hook, allowing you to switch to agent mode wherever you're.
K2 is one of the only models to nail the clock face test as well. It’s a great model.
Also shocking is how the most common runner up I've seen is DeepSeek
They are not just leeching here, they took this innovation, refined it and improved it further. This is what the Chinese is good at.
Muon paper: https://arxiv.org/abs/2502.16982
I wonder whether Claude is doing the same kind of training and it's coming with the next model, and that's why the agent swarm mode in Claude Code is hidden for now. We might be getting very very good agent orchestrators/swarms very soon.
You can of course "run" this on cheaper hardware, but the speeds will not be suitable for actual use (i.e. minutes for a simple prompt, tens of minutes for high context sessions per turn).
Not a "I actually use this"
The difference between waiting 20 minutes to answer the prompt '1+1='
and actually using it for something useful is massive here. I wonder where this idea of running AI on CPU comes from. Was it Apple astroturfing? Was it Apple fanboys? I don't see people wasting time on non-Apple CPUs. (Although, I did do this for a 7B model)
e.g. in an office or coworking space
800-1000 gb ram perhaps?
Interested in the dedicated Agent and Agent Swarm releases, especially in how that could affect third party hosting of the models.
Why is that Claude still at the top in coding, are they heavily focused on training for coding or is it their general training is so good that it performs well in coding?
Someone please beat the Opus 4.5 in coding, I want to replace it.
Also consider they are all overfitting on the benchmark itself so there might be that as well (which can go in either directions)
I consider the top models practically identical for coding applications (just personal experience with heavy use of both GPT5.2 and Opus 4.5).
Excited to see how this model compares in real applications. It's 1/5th of the price of top models!!
Sure it's SOTA at standard vision benchmarks. But on tasks that require proper image understanding, see for example BabyVision[0] it appears very much lacking compared to Gemini 3 Pro.
There aren't enough public domain data to create good LLMs, especially once you get into the newer benchmarks that expect PhD level of domain expertise in various niche verticals.
It's also a logical impossibility to create a zero knowledge proof that will allow you to attribute to specific training data without admitting to usage.
I can think of a few technical options but none would hold water legally.
You can use a Σ-protocol OR-composition to prove that it was trained either on a copyrighted dataset or a non copyrighted dataset without admitting to which one (technically interesting, legally unsound).
You can prove that a model trained on copywrited data is statistically indistinguishable from one trained on non-copywrited data (an information theoretic impossibility unless there exist as much public domain data as copywrited data, in similar distributions).
You can prove a public domain and copywrited dataset are equivalent if the model performance produced is indistinguishable from each other.
All the proofs fail irl, ignoring the legal implications, because there's less public domain information, so given the lemma that more training data == improved model performance, all the above are close to impossible.
Maybe we can get away with something cheaper than Claude for coding.
If a 10K computer could run this, it may be worth it to have a "fully on prem" version of ChatGPT running for you.
Also their license says that if you have a big product you need to promote them, remember how Google "gave away" site searche widgets and that was perhaps one of the major ways they gained recognition for being the search leader.
OpenAI/NVidia is the Pets.com/Sun of our generation, insane valuations, stupid spend, expensive options, expensive hardware and so on.
Sun hardware bought for 50k USD to run websites in 2000 are less capable than perhaps 5 dollar/month VPS's today?
"Scaling to AGI/ASI" was always a fools errand, best case OpenAI should've squirreled away money to have a solid engineering department that could focus on algorithmic innovations but considering that Antrophic, Google and Chinese firms have caught up or surpassed them it seems they didn't.
Once things blows up, those closed options that had somewhat sane/solid model research that handles things better will be left and a ton of new competitors running modern/cheaper hardware and just using models are building blocks.
Given the shallowness of moats in the LLM market, optimizing for mindshare would not be the worst move.
China has decided they are going to participate in the LLM/AGI/etc revolution at any cost. So it is a sunk cost, and the models are just an end product and any revenue is validation and great, but not essential. The cheaper price points keep their models used and relevant. It challenges the other (US, EU) models to innovate and keep ahead to justify their higher valuations (both monthly plan, and investor). Once those advances are made, it can be bought back to their own models. In effect, the currently leading models are running from a second place candidate who never gets tired and eventually does what they do at a lower price point.
The economist Mariana Mazzucato wrote a great book about this called The Entrepreneurial State: Debunking Public vs. Private Sector Myths
How many millions did Google spend on Android (acquisition and salaries), only to give it away for free?
Usually, companies do this to break into a monopolized market (or one that's at risk of becoming one), with openness as a sweetener. IBM with Linux to break UNIX-on-big-iron domination, Google with Android vs. iPhone, Sun with OpenSolaris vs. Linux-on-x86.
If you look at past state projects, profitability wasn't really considered much. They are notorious for a "Money hose until a diamond is found in the mountains of waste"
My biggest source of my conspiracy is that I made a reddit thread asking a question: "Why all the deepseek hype" or something like that. And to this day, I get odd, 'pro deepseek' comments from accounts only used every few months. Its not like this was some highly upvoted topic that is in the 'Top'.
I'd put that deepseek marketing on-par with an Apple marketing campaign.
In a few years, or less, biological attacks and other sorts of attacks will be plausible with the help of these agents.
Chinese companies aren't humanitarian endeavors.
https://openrouter.ai/moonshotai/kimi-k2-thinking https://openrouter.ai/moonshotai/kimi-k2-0905 https://openrouter.ai/moonshotai/kimi-k2-0905:exacto https://openrouter.ai/moonshotai/kimi-k2
Generally it seems to be in the neighborhood of $0.50/1M for input and $2.50/1M for output
If you’re really lazy - the quick summary is that you can benefit from the sweet spot of context length and reduce instruction overload while getting some parallelism benefits from farming tasks out to LLMs with different instructions. The way this is generally implemented today is through tool calling, although Claude also has a skills interface it has been trained against.
So the idea would be for software development, why not have a project/product manager spin out tasks to a bunch of agents that are primed to be good at different things? E.g. an architect, a designer, and so on. Then you just need something that can rectify GitHub PRs and bob’s your uncle.
Gas town takes a different approach and parallelizes on coding tasks of any sort at the base layer, and uses the orchestration infrastructure to keep those coders working constantly, optimizing for minimal human input.
I think the best way to think about it is that its an engineering hack to deal with a shortcoming of LLMs: for complex queries LLMs are unable to directly compute a SOLUTION given a PROMPT, but are instead able to break down the prompt to intermediate solutions and eventually solve the original prompt. These "orchestrator" / "swarm" agents add some formalism to this and allow you to distribute compute, and then also use specialized models for some of the sub problems.
then it creates a list of employees, each of them is specialized for a task, and they work in parallel.
Essentially hiring a team of people who get specialized on one problem.
Do one thing and do it well.
Where we have more specialized "jobs", which the model is actually trained for.
I think the main difference with agents swarm is the ability to run them in parallel. I don't see how this adds much compared to simply sending multiple API calls in parallel with your desired tasks. I guess the only difference is that you let the AI decide how to split those requests and what each task should be.
I looked through the harness and all I could find is a `Task` tool.
https://openrouter.ai/rankings
There are a lot of applications where you really just want a cheap and efficient model that's still somewhat competitive and that's exactly the niche DeepSeek fulfills the best.
Keep in mind that most people posting speed benchmarks try them with basically 0 context. Those speeds will not hold at 32/64/128k context length.
Anyway, in the future your local model setups will just be downloading experts on the fly from experts-exchange. That site will become as important to AI as downloadmoreram.com.
These are order-of-magnitude numbers, but the takeaway is that multi H100 boxes are plausibly ~100× faster than workstation Macs for this class of model, especially for long-context prefill.
Could be true, could be fake - the only thing we can be sure of is that it's made up with no basis in reality.
This is not how you use llms effectively, that's how you give everyone that's using them a bad name from association
Depending on how well you bargain with the robot, you can go as low as 0,99$ (difficult). Either way, their moderate plan doesn’t have to be 20$. The agent wants a good reason for why it should lower the price for you.
Here’s the direct link to Kimmmmy:
https://www.kimi.com/kimiplus/sale
I’ll send an invite link too if you don’t mind:
https://www.kimi.com/kimiplus/sale?activity_enter_method=h5_...
One positive side effect of this is that if subagent tasks can be dispatched to cheaper and more efficient edge-inference hardware that can be deployed at scale (think nVidia Jetsons or even Apple Macs or AMD APU's) even though it might be highly limited in what can fit on the single node, then complex coding tasks ultimately become a lot cheaper per token than generic chat.
My point was that this is just a different way of creating specialised task solvers, the same as with MoE.
And, as you said, with MoE it's about the model itself, and it's done at training level so that's not something we can easily do ourselves.
But with agent swarm, isn't it simply splitting a task in multiple sub-tasks and sending each one in a different API call? So this can be done with any of the previous models too, only that the user has to manually define those tasks/contexts for each query.
Or is this at a much more granular level than this, which would not be feasible to be done by hand?
I was already doing this in n8n, creating different agents with different system prompts for different tasks. I am not sure if automating this (with swarm) would work well in my most cases, I don't see how this fully complements Tools or Skills
Or did I misunderstand the concept of MoE, and it's not about having specific parts of the model (parameters) do better on specific input contexts?
Anyway, we were trying to run a 70B model on a macbook(can't remember which M model) at a fortune 20 company, it never became practical. We were trying to compare strings of character length ~200. It was like 400-ish characters plus a pre-prompt.
I can't imagine this being reasonable on a 1T model, let alone the 400B models of deepseek and LLAMA.
That said, I wouldn't necessarily recommend spending $20,000 on a pair of Mac Studios to run models like this. The performance won't be nearly as good as the server-class GPU hardware that hosted models run on.
Language is full of issues of particulars vs universals, and you could debate if its just an integrated GPU with different marketing.
Whatever the case, we couldn't use it in production, and NVIDIAs stock price reflects the reality on the ground.
My 2 cents
Scaling depends on hardware, so cheaper hardware on a compute-per-watt basis only makes scaling easier. There is no clear definition of AGI/ASI but AI has already scaled to be quite useful.
? Yes there's a clear defintion
https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
I think you mean there is no agreed upon definition? Even that is dubious statement.
So they are on the same page as the UN and US?
The One China policy refers to a United States policy of strategic ambiguity regarding Taiwan.[1] In a 1972 joint communiqué with the PRC, the United States "acknowledges that all Chinese on either side of the Taiwan Strait maintain there is but one China and that Taiwan is a part of China" and "does not challenge that position."
https://en.wikipedia.org/wiki/One_China https://en.wikipedia.org/wiki/Taiwan_and_the_United_Nations
That is why Chinese Taipei is not independent and why Beijing has said that if they threaten independence (that is to say, that if Chinese Taipei relinquishes its claim over the mainland -- that it in a way secedes from the union state) that the mainland would move to take full political control over the island.
It's a highly unusual situation -- there are countries which have governments in exile, this is not that. You can think of this like one country with two separate competing governments which irreconcilable claims. Clearly given the mainland's economic (and consequent military rise) Chinese Taipei cannot take over the mainland and again considering the power differential they would be insane to declare full independence. Why is why this bizarre situation persists.
Furthermore third-party countries which recognise Chinese Taipei's independence is shrinking and shrinking (and so it should be) because properly speaking the sane thing to do is recognise that there is only one China (because both the the authority of mainland China (the CPC) and the government of Taiwan assert that there is but one China) but deal with each authority separately. Economically speaking and politically speaking this is how things play out.
Because the situation is so unusual I think it is fair to say that most people just assume Taiwan is a breakaway province. And it's not surprising that they do because that's what happens normally in these situations. That would be the case if Taiwan amended it's constitution to explicitly relinquish its claim over the mainland but they are not going to do that unless they are prodded to by an outside third-party or if they suddenly get a wave of nationalism or if they suddenly become collectively suicidal. So this highly unusual and bizarre status quo persists.
This is my interaction with ChatGPT (obviously it would be pointless talking to DeepSeek, etc about this)
===
ME: does chinese taipei lay claim to mainland china?
ChatGPT: Yes — but it depends on *which “Chinese Taipei” you mean*:
### \*Chinese Taipei as the Republic of China (Taiwan)\*
\* The government based in Taiwan is officially the \*Republic of China (ROC)\*. Under its \*old constitution\* (dating back to when the ROC ruled all of China), it *theoretically* still contains language implying sovereignty over all of China — including the mainland. In practice, modern Taiwanese governments (especially under the Democratic Progressive Party) do \*not actively assert a real claim to govern mainland China\* and focus on governing Taiwan and nearby islands. ([House of Commons Library][1]) ### \*People’s Republic of China (PRC) claims Taiwan\*
\* The \*PRC\* (mainland China) claims that Taiwan is part of China and should eventually be “reunified,” insisting on a \*One-China principle\* that includes Taiwan under its sovereignty. This is the basis of most countries’ diplomatic relations with Beijing instead of Taipei. ([House of Commons Library][1])
### Key nuance
\* Taiwan (the ROC government) effectively functions as a \*separate political entity with its own government, military, and democratic system\* and does \*not control or claim to govern mainland China’s territory in practice\* even if older constitutional wording could be interpreted that way. ([House of Commons Library][1])[1]: https://commonslibrary.parliament.uk/research-briefings/cdp-... "International status of Taiwan - House of Commons Library"
[2]: https://www.taipeitimes.com/News/front/archives/2025/03/08/2... "Taiwan rejects China’s UN claim - Taipei Times"
ME: What does "it theoretically still contains language implying" mean? It either does or it doesn't.
ChatGPT: You’re right to push back — that phrasing was imprecise. Let’s be exact.
### Short answer
\*Yes, the Republic of China (ROC) constitution *does* still claim all of China, including the mainland.\* ### What the constitution actually says
\* The \*ROC Constitution (1947)\* was written when the ROC claimed to be the sole government of *all* China. So legally speaking:
> \*Mainland China remains part of ROC territory under the constitution.\*
### Where the confusion comes from
The confusion arises because of \*later constitutional add-ons\*, not because the claim disappeared.
Since the 1990s, Taiwan adopted the \*Additional Articles of the Constitution\*, which:
\* Suspend application of many provisions *with respect to the mainland* Crucially:
\* These articles \*do not renounce the mainland\* So the situation is:
| Aspect | Status |
| ---------------------------------- | ------------------------------------- |
| Constitutional territorial claim | \*All of China (including mainland)\* |
| Practical governance | \*Taiwan and nearby islands only\* |
| Active effort to reclaim mainland | \*No\* |
| Formal constitutional renunciation | \*No\* |
### Why people hedge with language
People (especially diplomats and scholars) often say things like *“theoretically implies”* to avoid the blunt conclusion that:
\* \*Taiwan’s constitution still claims mainland China\*, which is politically awkward given Taiwan’s current identity and international position.
But legally, there’s no ambiguity:
\* \*It either does or it doesn’t — and it does.\*
### One-sentence summary
\*Chinese Taipei (the ROC) constitutionally claims mainland China, but politically and practically does not pursue or assert that claim.\*
If you want, I can also explain \*why Taiwan has never amended the constitution to drop the claim\*, because that’s a fascinating mix of law, identity, and geopolitics.
(Sorry for the dreadful formatting). I am more than happy to be corrected on this.
This dark side of all this is that the original inhabitants of the island are not "Chinese" (Han) and the island has had both Japanese and Chinese rule if I'm not mistaken: https://en.wikipedia.org/wiki/Taiwanese_indigenous_peoples
You can't with Tiananmen square in China
The Tiananmen Square killings happened independent of one's "worldview".
What's an example of "the exact same thing" done to OpenAI models by western governments?
EDIT: oh its a marketing account and the program never finished... who knows the validity.
The tooling involved has improved significantly over the past year.
Back when 4k movies needed expensive hardware, no one was saying they could play 4k on a home system, then later mentioning they actually scaled down the resolution to make it possible.
The degree of quality loss is not often characterized. Which makes sense because it’s not easy to fully quantify quality loss with a few simple benchmarks.
By the time it’s quantized to 4 bits, 2 bits or whatever, does anyone really have an idea of how much they’ve gained vs just running a model that is sized more appropriately for their hardware, but not lobotomized?
int4 quantization is the original release in this case; it's not been quantized after the fact. It's a bit of a nuisance when running on hardware that doesn't natively support the format (might waste some fraction of memory throughput on padding, specifically on NPU hw that can't do the unpacking on its own) but no one here is reducing quality to make the model fit.
The broader point remains though which is, “you can run this model as home…” when actually the caveats are potentially substantial.
It would be so incredibly slow…
Any model that I can run in 128 gb in full precision is far inferior to the models that I can just barely get to run after reap + quantization for actually useful work.
I also read a paper a while back about improvements to model performance in contrastive learning when quantization was included during training as a form of perturbation, to try to force the model to reach a smoother loss landscape, it made me wonder if something similar might work for llms, which I think might be what the people over at minimax are doing with m2.1 since they released it in fp8.
In principle, if the model has been effective during its learning at separating and compressing concepts into approximately orthogonal subspaces (and assuming the white box transformer architecture approximates what typical transformers do), quantization should really only impact outliers which are not well characterized during learning.
If this were the case however, why would labs go through the trouble of distilling their smaller models rather than releasing quantized versions of the flagships?
Fwiw, not necessarily. I've noticed quantized models have strange and surprising failure modes where everything seems to be working well and then does a death spiral repeating a specific word or completely failing on one task of a handful of similar tasks.
8-bit vs 4-bit can be almost imperceptible or night and day.
This isn't something you'd necessarily see playing around, but when trying to do something specific
IMO 1tln parameters and 32bln active seems like a different scale to what most are talking about when they say localLLMs IMO. Totally agree there will be people messing with this, but the real value in localLLMs is that you can actually use them and get value from them with standard consumer hardware. I don't think that's really possible with this model.
- have an i5 or better or equivalent manufactured within the last 5-7 years
- have an nvidia consumer gaming GPU (RTX 3000 series or better) with at least 8 GB vram
- have at least 32 GB system ram (tested with DDR4 on my end)
- build llama-cpp yourself with every compiler optimization flag possible
- pair it with a MoE model compatible with your unified memory amount
- and configure MoE offload to the CPU to reduce memory pressure on the GPU
then you can honestly get to about 85-90% of cloud AI capability totally on-device, depending on what program you interface with the model.
And here's the shocking idea: those system specs can be met by an off the shelf gaming computer from, for example, Best Buy or Costco today and right now. You can literally buy a CyberPower or iBuyPower model, again for example, download the source, run the compilation, and have that level of AI inference available to you.
Now, the reason why it won't work on Linux is that the Linux kernel and Linux distros both leave that unified memory capability up to the GPU driver to implement. Which Nvidia hasn't done yet. You can code it somewhat into source code, but it's still super unstable and flaky from what I've read.
(In fact, that lack of unified memory tech on Linux is probably why everyone feels the need to build all these data centers everywhere.)
So it should work with an AMD GPU?
Depends on if AMD (or Intel, since Arc drivers are supposedly OSS as well) took the time to implement that. Or if a Linux based OS/distro implements a Linux equivalent to the Windows Display Driver Model (needs code outside of the kernel and specific to the developed OS/distro to do).
So far, though, it seems like people are more interested in pointing fingers and sucking up the water of small town America than actually building efficient AI/graphics tech.
There is a huge difference between "look I got it to answer the prompt: '1+1='"
and actually using it for anything of value.
I remember early on people bought Macs (or some marketing team was shoveling it), and proposing people could reasonably run the 70B+ models on it.
They were talking about 'look it gave an answer', not 'look this is useful'.
While it was a bit obvious that 'integrated GPU' is not Nvidia VRAM, we did have 1 mac laptop at work that validated this.
Its cool these models are out in the open, but its going to be a decade before people are running them at a useful level locally.
At 3 tk/s, that's still 100-150 pages of a book, give or take.
If it were 2016 and this technology existed but only in 1 t/s, every company would find a way to extract the most leverage out of it.
But to answer your question directly, tensor parallelism. https://github.com/ggml-org/llama.cpp/discussions/8735 https://docs.vllm.ai/en/latest/configuration/conserving_memo...
LLMs which the weights aren't available are an example of when it's not local LLMs, not when the model happens to be large.
I agree. My point was that most aren't thinking of models this large when they're talking about local LLMs. That's what I said, right? This is supported by the download counts on hf: the most downloaded local models are significantly smaller than 1tln, normally 1 - 12bln.
I'm not sure I understand what point you're trying to make here?
Over any task that has enough prefill input diversity and a decode phase thats more than a few tokens, its at least intuitive that experts activate nearly uniformly in the aggregate, since they're activated per token. This is why when you do something more than bs=1, you see forward passes light up the whole network.
I was trying to correct the record that a lot of people will be using models of this size locally because of the local LLM community.
The most commonly downloaded local LLMs are normally <30b (e.g. https://huggingface.co/unsloth/models?sort=downloads). The things you're saying, especially when combined together, make it not usable by a lot of people in the local LLM community at the moment.
With distillation you're training new model, so size of it is arbitrary, say 1T -> 20B (50x) reduction which also can be quantized. AFAIK distillation is also simply faster/cheaper than training from scratch.
"Never attribute to malice that which is adequately explained by stupidity."
Yes, I'm calling labs that don't distill smaller sized models stupid for not doing so.
That doesn't mean training with all integer math, but certain tricks are used to specifically plan for the end weight size. I.e. fake quantization nodes are inserted to simulate int4.
You're of course accurate that smaller LLMs are more commonly deployed, it's just not the part I was really responding to.
"Secure" in the sense that they can sue someone after the fact, instead of preventing data from leaking in the first place.
Thing is, people in the local llm community are already doing that to run the largest MoE models, using mmap such that spare-RAM-as-cache is managed automatically by the OS. It's a drag on performance to be sure but still somewhat usable, if you're willing to wait for results. And it unlocks these larger models on what's effectively semi-pro if not true consumer hardware. On the enterprise side, high bandwidth NAND Flash is just around the corner and perfectly suited for storing these large read-only model parameters (no wear and tear issues with the NAND storage) while preserving RAM-like throughput.
I can run Minimax 2.1 in 5bpw at 200k context fully offloaded to GPU. The 30-40 tk/s feels like a lifetime for long horizon tasks, especially with subagent delegation etc, but it's still fast enough to be a daily driver.
But that's more or less my cutoff. Whenever I've tested other setups that dip into the single and sub-single digit throughput rates, it becomes maddening and entirely unusable (for me).
e.g. A 4-bit quant can have half the attention and feed forward tensors in Q6, and the rest in Q4. Due to how block-scaling works, those k-quant dtypes (specifically for llama.cpp/gguf) have larger bpw than they suggest in their name. Q4 is around ~4.5 bpw, and Q6 is ~6.5.