The Kimi K3 Moment(stephen.bochinski.dev) |
The Kimi K3 Moment(stephen.bochinski.dev) |
But damn Moonshot.
Kimi K3: Open Frontier Intelligence
https://news.ycombinator.com/item?id=48935342
Kimi K3, and what we can still learn from the pelican benchmark
"...I’ve been running Kimi K3 alongside Claude on my normal coding work, and for all practical purposes I can’t tell them apart. Same tasks, same quality of output, and near identical token counts to get there. I expected an open model to be sloppier or to grind through more tokens on the way to the same answer, and neither turned out to be true.
The prices are nowhere near each other. K3’s API runs $3 per million input tokens and $15 per million output. Claude’s top model costs $10 and $50 for the same units. The subscription side is even more lopsided..."
> When the headline model on your plan can be switched off because the economics don’t work, the plan was never really selling you the headline model. Kimi’s tiers don’t come with that asterisk.
This line has a certain smug, punchy cleverness that I associate with AI. To me, the vibes are ~30% AI writing.
In a few years there will be Mythos level open weight models hosted by the lowest bidder anyway.
At the rate things are moving I'd expect that to happen much sooner.
In fact: Somebody, right now as we speak, is most likely already working on training the next best open source model.
I just thought about that recently too, then Kimi K3 came out, and I thought: Yea, I'm not surprised. Just a matter of time now...
And this is the point where your internal compiler should have started shouting 'Type Error'
Notice the trick here?
> Then there’s the fine print. Claude couldn’t sustain Fable access on the twenty dollar plan, so they turned it off, and the plan quietly falls back to Opus.
Where is the Fable-class Kimi model at all?
i posted the transcript as a reply to you, and HN automatically flagged it.
but go on.
Even if it didn’t happen here, it was still the case that it was going to happen going forward. It was always going to end like this. Invest in the hardware companies, not the model companies.
Thus far the US has not really chosen to go the Chinese rare-earth method yet. The problem with distillation attacks is the end result is everyone who is not doing them is going to deal with some kind of regulation whether it's complete loss of access, or the amount of control you'll have to give up to access them will be ridiculous.
Sort of like the "stealing music is fine" but "lets freak out now that it's producing visual art", in the end the entire thing is a social construct. Whether this is treated as theft or "business as usual" is entirely societal.
Eventually the gap will close, unless there's a major breakthrough that hasn't been made yet.
I'm not aiming for a what about kickflip here: I'm saying we need to either agree on some rules or stop crying foul. Maybe the coherent legal theory is that neural networks and intellectual property don't interact. That would be weird but it would be consistent, a market could price it, I could do coding stuff and know if I was illegaling.
But this weird gerrymander that no judge will really rule on in an emphatic way is like, bad for the planet, bad for markets, bad business.
There are a lot of reasons to look forward to DeepSeek Huggingface drop kicking the unambiguous frontier weights in like, November, but I think my favorite one will be "who's distilling now bitch?"
But that said: K3 is not a distilled version of Fable or Sol. Fable has been barely available and Sol was just released! Moreover, K3 is superior to both models in some domains, according to user scoring on the Arena.
API distillation can’t give you these results anyway. All it is useful for is bootstrapping RL in new domains to get past the “cold start” problem faster. By far, what matters more is the quality and variety of RL environments the model learns from.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc._v._Ru....
And then there's new updates related to AI that fully take out LLMs from protection.
These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs.
Setting aside the fact that it doesn’t make any feasible sense to do API distillation, these models are outperforming frontier models on a number of benchmarks, and often times run more efficiently by several orders of magnitude.
We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.
yeah hardware companies make for nice stories or green numbers on Wall Street - but value will be captured by application layer.
look at history.
what is the end game for this strategy?
if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress?
I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.
So why didnt we have these LLMs in 2005?
Don't forget to account for all the costs. It's not just that CPUs are X times slower. Memory is X times smaller, too, and networks are X time slower. And all this hardware is many times more expensive.
If I'm getting my mental estimation right, training a 2026-frontier-class LLM in 2005 would be somewhere on the order of all the computation power in the world at the time. It's not that many more factors of magnitude before you end up at "all the computation power in the world up to that point".
1: That's the "T" in GPT fyi, even though Google is the author of the research paper that changed everything
The first "L" in LLM does the work. In 2005 you had no Github, Stackoverflow, Youtube, common crawl and no archive of digital ebooks.
If "distillation attacks" happen, we have to conclude there is some value add in what model labs do. Regardless of how we feel about using existing human knowledge in the way they currently do, it's simply impractical to infer that everything that happens downstream of LLMs can not be an attack on some IP because of it.
So both things can be true: a) People infringe on Anthropics IP and b) what Anthropic did to build their models is legally questionable (or might be ruled illegal, even though I doubt it).
No.
Authors do not infringe on IP when they read another's book, nor should the lumber company be able to dictate how I use planks and if I can resell them if i'm done with them.
You're framing it as if the added value of the author or lumber company, awards them consideration when somebody uses the products to create more value.
IP law was always a big mess, and these questions cross far into ideology instead of law; but I do not understand people who think we need an ideology where more IP-law is good for society.
Unless someone literally stole the weights somehow (which is not out of the question, I doubt either oAI/Anthropic have the capabilities to prevent a state-level actor getting those weights), distillation from generations is not infringement on anyone's IP nor is it stealing nor is it an attack. It can't be. As long as you pay for tokens you get to do whatever you want with them. Someone saying you can't doesn't mean it's an attack or their IP or whatever. They either sell the tokens or not. They can decide to not sell them to anyone, but again that's not stealing.
And their ToS are a joke. Imagine how people would react if MS had ToS saying that you can't use MS software to develop solutions that compete with MS. They'd be laughed out of the room. Somehow it's ok for token sellers to decide what you do with the tokens? Why? If you pay for something you get to do whatever you want with that output. Train, distill, whatever.
Anthropic’s model outputs contain no IP. This is actually a simple legal proposition (rare in this field!) that derives from the fact that only specific classes of IP exist: copyrights, patents, trade secrets, and trademarks. Examining each, it is clear that API outputs do not qualify. Anthropic disclaims copyright in outputs; the outputs are not patented; the outputs are not secret (a prerequisite to having trade secrets); and trademarks are irrelevant in concept.
Anthropic, OpenAI, etc do not deserve legal protection.
that would be existential doom for them because then they have a case to claim ownership of their users' codebases
no corporation would sign off on that
Do you really think intellectual property laws will prevent this in practice? It’s like as if we said, “hey, USSR, you can’t make a nuke, too! We patented that already.”
Asking China to not distill our models down is equally as ridiculous.
I only have the $20 plan from OpenAI and the same task, with a lot of the same implementation details as Kimi Code, only took a few minutes and consumed almost none of the 5 hour limit.
Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare, but when I sat down to add Kimi Code to flar, it was because I wanted to try it on some real work and then couldn't do any, because usage was nearly gone after the trivial task...no other ~$20 subscription I have has felt that tight before.
So, it was really slow to complete the task and seemingly much more expensive than every other model I'd tried. Maybe bad luck. Maybe it'll do better on other tasks. I wouldn't know as I was out of usage when I had time to try.
It did find a bug that Gemini 3.5 Flash introduced unprompted, though, so it has that going for it.
Once western governments declare it to be a "national security" risk for citizens to have access to open-weight frontier models, and once they classify using these models as acts of terrorism, what will that world be like?
Will using Kimi K3 come to be like how napster was in the olden days? Everybody knew it was technically illegal, but come on -- any track at your fingertips? But surveillance is quite more evolved now.
Or it will be like cannabis, where a guy in the neighborhood will low key rent you metered access to the 8x5090 rig in his basement he cobbled together from parts on ebay? Or everyone will flock to VPNs?
Or will the oppressors actually succeed? The same way that napster is long gone, and everyone accepts that they must pay spotify for a homogenized collection, where artists must take only a minuscule cut (more than napster though)... We'll be stuck with nerfed Cohere or Mistral models for open-weight options, as if they need more lobotomizing. Or else we can pay through the nose for Anthropic/OpenAI for "American Frontier" models which will fall increasingly far behind China.
Or else, like how Kindle Fire was subsidized by ads, we'll have "Kindle AI" where influence is sold to the highest bidder, where the LLM will tell us that smoking is actually healthy if big tobacco can engineer its renaissance by turning its lobbying dollars to pay-to-play, pumping its propaganda into the training pipeline for Amazon's extra commercialized line of ultra budget LLMs.
Kimi K3 has 2.8 trillion parameters. We don't know the number of parameters of ChatGPT 5.6 or Opus 4.8, but it's probably in the same region. Fable/Mythos are rumored to be around 10 trillion.
So, K3 is directly comparable with ChatGPT 5.6 and Opus 4.8, and the price is not so much lower:
K3: $3/$15 per 1 Mtok input/output ChatGPT 5.6 Sol: $5/$30 Opus 4.8: $5/$25
This is not a watershed moment. It's a competitor converging to the same capability and trying to undercut your prices, but not by a lot.
As for the open weights? For now, Kimi K3's weights are closed, and I don't expect the situation would change.
It'll change on July 27 (based on https://www.kimi.com/blog/kimi-k3):
> The full model weights will be released by July 27, 2026
This does depend heavily on the kind of work you do and how you use these models, but the idea that K3 isn't right up there with US SOTA models doesn't match my experience.
That's weeks maybe months behind, not months maybe a year behind. It's "would my life really change if Claude was gone, not really" behind.
I actually haven't used it much, because Claude started kicking ass again the last few days. Like, way too much of a difference to be normal load-based variance. I got more done in the last 48 hours than week before that.
So, fuck yeah competition.
If you sign up with non-Chinese phone number, you're bucketed into US, you get US prices, can pay only in USD and with American credit card network.
Chinese prices are about 9x cheaper than the US prices, which are already far cheaper than Claude or other American provider. If you can somehow get hold of a Chinese phone number, keep in mind that you can save ~90% of the bill.
My assumption is that Anthropic, OpenAI, Kimi, etc all have a similar cost structure when serving models. The same size model roughly generates the same GPU usage whether you’re American or Chinese. I’d also guess that the model sizes across all SOTA models is similar, we just only see data for open models. The difference is most likely that American companies simply charge more because they have the dominant market position.
Remember not too long ago when Anthropic was charging $75/mt for Opus? Now that many models are in “opus tier”, their pricing is $25 - higher than competitors but close. The newest Kimi is $15. 40% lower to forgo “made in America” with American enterprise support staff is not crazy. Compare AWS to Hetzner or any other flagship enterprise service to the foreign and discount option. I assume that over time, we’ll see the commodification of models reducing prices even towards the raw GPU costs.
When you say "Claude", do you mean Opus? Fable? What effort level?
Those things do make a difference to some of us, even though nothing is black and white. In my case, I'll probably want to wait until other providers appear through OpenRouter and then I'll try to judge how much I trust them. But even if I don't trust them much, they don't train models anyway, so the likelihood of my data being used that way is smaller.
I find these kinds of concerns increasingly silly: most of the input to these models will be ... previous output from the very same models, alongside the occasional half-assed human command to fix something and "make zero mistakes". Who cares if they train on that? Let them, if it makes their future models better!
99% of users are not working on any special IP to worry about that.
For example, Kimi 2.7 has been really effective for me despite having verbose thinking blocks, simply because it runs so fast. Speed-wise, it feels about like Sonnet, possibly faster.
Can’t use for commercial purposes. Can’t opt out of training. Data retained.
"Can't use for commercial purposes" - incorrect AFAICT. In what sense do you mean this? The open weight MIT version obviously allows for commercial use, but I don't think that's what you're referring to, because training data is irrelevant on the open weight version. Pretty sure the API allows commercial use too. Maybe the free version doesn't? But who cares?
Is anyone using open source models for anything major ?
I don't understand how a product that:
- is interfaced with and is deeply linked to natural language, so everything you produce (sessions, history, etc) is in Markdown and you can literally install a second model and tell it "hey import all of Claude's memory into yours" and that's it
- is based on well understood technology, the real constraints are how much money you put into training the models, but the theory has all been developed in the open
- clearly has a threshold where it quickly commoditises and turns from "I want the best" to "hey the best is a bit too expensive. The second best is half the price and works close enough".
was ever supposed to be a money printing machine. The fact something is extremely useful doesn't imply it's extremely profitable.
IMHO we're clearly speedrunning the process of turning AI into a commodity. Dario Amodei knows pretty well that when or if Anthropic cuts people off Fable, the vast majority of them will definitely not pay for it because Opus 4.8 is good enough for almost everybody that _knows_ what they're doing, and so are basically half of the most recent models. If I already have good baking skills I don't become more productive with an automatic bread machine, I just need a better dough mixer and oven
What is the parento frontier?
Here's the thing about this though, the auto industry directly employed hundreds of thousands of people.
The AI labs are small, only few benefit directly from their wealth and there's already immense opposition to AI, data centers, etc...
Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that.
Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so
I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences.
I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.
AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.
Now, of course, the plan is to remove Fable from the subscription. To paraphrase Darth Vader, they have altered the deal. Pray they do not alter it further.
What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla.
Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories.
What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost.
I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.
Subsidies would affect 1, but not 2. But if some VC wants to subsidize my Claude or Codex or whatever, awesome.
ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty.
> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare
I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use.
In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai.
[0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.
The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems.
There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe.
There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest.
If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.
So maybe some isolated switzrland/singapore type locales would exist for US/EUusers to be able to dip their toes across the curtain legally without reprucursions.
[1] https://nitter.net/RnaudBertrand/status/2069574934972797089
If you need infrastructure done, China is dominating that area too. Rail, High-speed rail, Nuclear reactors, (near future Thorium reactors), Dams, Highway roads, bridges, Ocean ports, airports you name it, and they can roll it out, Transport ships, And if they don’t do it, Japan, Korea, Vietnam, and Taiwan do.
Is it too late? No, not necessarily, but America needs a regime change…
Are you talking about the US, specifically?
Why would other countries, that don't share the same anxiety about China as the US, would be troubled with the this?
That may have been closer to reality 10-20 years ago, China is a different country now, what I mean by that is they offer research funding, they have huge digital behemoths (alibaba, tencent, huawei, bytedance etc), large scale deployment opportunities and prestigious careers. Many graduates return because the opportunity set is attractive and they want to return, it's not just because US immigration policy pushed them out. Some also want to contribute to their own country's technological progress (which is a normal motivation btw), like probably you are also a patriot and want your country to succeed.
So, really, China's AI progress is not mainly the result of America failing to absorb every talented Chinese researcher. China has built a domestic ecosystem capable of producing and keeping top talent itself. I feel like a lot of Americans do not understand this.
[1] https://www.latimes.com/world-nation/story/2025-02-21/why-ch...
[2] https://www.wsj.com/world/china/americas-allure-fades-in-chi...
[3] https://www.theguardian.com/world/2025/jun/06/chinese-studen...
US immigration policy isn't a big factor.
China's got 1.8B people. If you don't think they've got the talent to pull this off, even if a lot of it leaves to live elsewhere, you're naive.
No one uses Baidu, but they built their own Google, and it's good.
They built their own Facebooks and Instagrams.
The US isn't the only place in the world where people can build software...
20k O-1 visas were issued last FY which was mostly under the Trump admin, up from 19.5k the previous FY under the Biden admin
When people demonstrate their capability thoroughly, the Chinese government takes away their passports. You’re not exactly going to get them here with an O-1.
- china’s homegrown tech industries already achieved escape velocity from it a long time ago, after China fenced off its market for Alibaba and Baidu in the ‘00s. some of their AI innovation at the edges was already top class 10 years ago
A closely related question is “what do the American labs need to do in order to justify their enormous market valuations?”
It seems like the answer cannot possibly be “gradually improve model capability while figuring out how to better monetize inference.” The valuations are just way too high for that to be sufficient.
Surely the answer has to be “continually achieve large leaps in capability comparable to the first consumer releases of ChatGPT while also maintaining a significant capability lead over open models and new competitors.”
And does anyone think that’s going to happen? Even with state-level protection from competition (which incidentally would significantly harm the American economy), the large leaps in capability seem to be coming fewer and farther between.
What appeared initially to be a huge innovation was later easily duplicated by many. There are no platform-lockins or network effects. Switching costs for users are zero, and there are low barriers to entry, with vast numbers of models to choose from and more appearing every day. As a business a token will be a commodity like an electron. Doesnt matter who produces it, or how (solar, wind, coal, nuclear etc) as long as it powers my toaster.
everything else we see today is just preparing for it.
So the initial models arent just distilled from information. We’ve always had the information.
(I actually appreciated your analogy, despite my lark)
The Pareto frontier tells you which designs are the best in at least one of your metrics (non-dominated by another design). For example if you're selecting a car and you care about both speed and mpg, a Formula 1 car and a Prius might lie on the Pareto frontier, but a Model T Ford would not.
So like, on a cost-intelligence graph, the cheapest and most intelligent models are pareto optimal. Then in-between those if you have
- cost $3 intelligence 6
- cost $1 intelligence 5
- cost $2 intelligence 4
The 1st and 2nd are pareto optimal, the 3rd is not, because it's dominated by the 2nd (2nd is cheaper AND more intelligent at the same time)
Additionally that same VC could be (read: is always) spent on developing the harness, and other infrastructure around the model, not just the model itself.
So it's apples-to-oranges when comparing a relatively new model to established competitors (i.e. OpenAI @ $900B funding vs Moonshot/Kimi's $30B FYI) because every new model they release is judged on "performance" which is not strictly speaking derived solely from the model.
It's possible Moonshot could get similar performance over time as the build out the rest of the infrastructure. We have no way of knowing how much of OpenAI/Anthropic's success is due to the model vs intelligent tooling built on top of it.
It’s going to be a different world, a world where many former allies are not gonna look to the United States first they can no longer afford to.
It has nothing to do with running open models, especially in hardware within Europe.
Basically of all visas O-1 is virtually guaranteed to have highly positive economic value
How do things get "established" from someone's perspective, exactly?
By that logic it is established from my perspective that Anthropic has no right to train on anything I've written that is publicly available on the internet.
Of course, they don't care about my perspective, but then again I don't care about theirs.
It's too late because the US needs a population change. The red states are leeches and rural people are the precise kind of undesirable they smear immigrants as.
America is what it is. The only thing that will change it is leaning in not bemoaning "rural people" on Hacker News.
And, DeepSeek is what I use for any task that works best with an API. It's cheap enough to where I don't think about cost, made even cheaper by DeepSeek having the most effective and cheap caching in the industry, and it's good enough to where I rarely have to follow up with a more expensive model or manually fix things. It's been alleged they're releasing an update to DeepSeek V4 Pro soon that improves it, which likely makes it a good fit for even more kinds of problems. It remains my favorite of the Chinese models, it's so cheap and cheerful. And, is also less aggressively censored than some of them.
I use DeepSeek v4 Flash & MiMo v2.5 Pro. Prefer the latter over DeepSeek v4 Pro because it costs the same while being equally good & less chattier for coding workloads. Although, I've begun experimenting with Hy3 (as an in-between Flash & Pro) & GLM 5.2 (for long-horizon tasks).
On Openrouter Kimi K3 says it does not retain data or train on it, which is better than what US hosts claim for Claude, ChatGPT, etc.. as they collect and retain data even if you disable training on it.
Opencode or similar open source tool + a zero data retention provider is about the best option aside from running a smaller fully local model on your own PC.
I guess you could steal them but thats a whole other issue.
Are the distillers reading books or are they building models?
If anthropic is providing no value they can just build from scratch. But obviously distilling is easier. Hes saying thats the value they add.
This would demolish agent usage by corporations.
if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off?
I see no evidence for that.
> if this were not the case, then we would be observing chinese models that far surpass frontier models
It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI.
> what happens to these efforts when the subsidy is cut off?
They will continue making progress as they do now, minus the benefits of distillation.
Moonshot AI Scale: Over 3.4 million exchanges
The operation targeted:
Agentic reasoning and tool use Coding and data analysis Computer-use agent development Computer vision Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.
Under the settlement, Anthropic was forced to delete the pirated data they were training on.
Chinese labs can still train on pirated data. I doubt the Chinese models operate under similar licensing agreements.
The payment was for illegally downloading copyrighted material, not training. Training was explicitly ruled to be fair use.
> I doubt the Chinese models operate under similar licensing agreements.
US corps likely pay licenses when afraid to be sued, or have troubles getting that data, otherwise they just take data, which was demonstrated many times. The same apply to Chinese corps, alibaba totally can be sued in US.
there is much less intellectual property in China so it’s not ‘theft’ (as you can’t put property on information)
and what will fund these budgets exactly? inference is cheap, distillation is cheap, training is what's expensive.
"Distillation" literally means to separate and take some components out of something. You can distill how a model works from a model. You cant distill a model from information because the information does not contain the model.
People are happy to conflate distilling with building because they dont like how the information was used. You distill how the model works from the model, and you build a model with information. Both could be morally good or bad but its not the same thing.
Not really, what the information actually is, matters a great deal. It's harder to get good results going from "nothing > model+weights" than "nothing + traces from known good sessions of other good model > model+weights", this is what the "distillation" part is referring to. If "information is information", you wouldn't even need to separate good from bad sessions while doing the training, which leads to somewhat obvious results if you don't.
To succinctly restate my point, you cannot distill a model from information because the model is not contained within that information. You can distill a model from another model.
Training on legally acquired / licensed data is potentially fair use.
Translation: we have the machinery in place to identify our users, and actively do so.
But the American AI companies only let you query their models if you first sign a contract to not train on the output.
It's hypocrisy and unfair, but I think there's a strong legal argument for it.
Of course China can simply decline to assist in enforcing that contract... But I would expect US courts to do their best to.
Now THAT'S doing some heavy lifting lmao. The vast, vast, VAST majority of the original datasets were from pirated books and the like. Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing, yet the AI cos choose time and time and time again to simply ignore it and be as abusive as they possibly fucking can
And there's been significant legal consequences as a result
> Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing
You're free to argue this of course, but the courts have largely rejected it already pre LLMs. See for example hiQ Labs v. LinkedIn
That right there is the problem.
Maybe today. I doubt it tomorrow. Legal and not legal, largely, has to answer to the population sooner or later. Ultimately, humanity decides legality. And I don't think the frontier labs will get a pass from humanity in the midterm, let alone the long term. I think you'll see the rules change towards something more "intent" driven. And there's absolutely no difference in intent between Frontier labs and everyone chasing them.
Frontier labs just want the door closed behind them, as do their investors, because they know the money will never be recouped if others can do the same magic tricks.
Or do you just mean that US courts don't have enough teeth to prevent Chinese companies from violating contracts? On that I agree.
Situation right now seems more like a fragile detente: if you got a Hill staffer drunk and hounded him long enough he'd probably be like "God damnit the market will fucking tank if we don't get these two IPOs out north of a trillion. And don't even get me started on how I'm going to sell Chinese AI to a Senate that still calls people Nipponesians when no one is looking. We're doing the best we can alright, get off my back man."
We have a situation, but it's not exactly A&M Records, Inc. v. Napster.
Oh it is, and at least anthropic has paid $1.5 billion and deleted there torrented copies and not released any models derived from them as a consequence.
The thing is it turns out to be not that expensive to just buy a copy of every book legally and scan them. And there's even precedent that this is legal predating LLMs (Google books)
Granting people some form of control over knowledge only serves the public interest inasmuch it provides incentive to create more of it. Mass media, effortless duplication, and copyright extensions had already broken this to the point where control of knowledge was suppressing creation of new knowledge more than it facilitated.
The world has changed, we need a mechanism that works for the public interest that applies to the facts as they now are.
The US is already publicizing the way they are using Claude with Palantir for war gaming purposes. It’s a matter of national defense. Contract law has no meaning here.
I have a bridge to sell you
Facts are in fact knowable, and the US legal system is in fact not terrible at getting to them.