Thanks, I'll try it! I guess "mixing" GPU+CPU would hurt the perf tho.

They're keeping some stuff to themselves which is fine. I don't expect anyone to have to fully release everything they've got especially considering the vast costs associated with researching and developing these models.

What they have released has been distilled into many new models that others have been using for commercial benefit and I appreciate the contributions that they have made.

you can do these same, except you would need to be a pirate website. It would even be better. except illegal. but it would be better.

0.5% or 0.03% satisfy my "nonalcoholic" criteria.

> Studies have found ethanol levels in commercial apple juice ranging from 0.06 to 0.66 grams per liter, with an average around 0.26 grams per liter[1]

Even apple juice is an alcoholic drink if you push your criteria to absurdity.

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC5421578/

but they're not bleach, and no amount of adding or removing alcohol can transmute the alcohol into something else.

That’s the ‘source’ for what the model spits out though, if not the source for what spits out the model.

:) The good ol days!

It's a useful diagnostic when used in a battery of diagnostic tests of cognitive function, but to the point of this thread: it is notoriously not a good ranking mechanism.

Thank you, but what do you use the llm for? Writing new documents based on previous ones? Tagging/categorization/summarization/lookup? RAG? Extracting structured data from them?

Most of those items map to a job description.

If you think the data story isn't a complicated beast, then consider:

If you wanted an "open" dataset, would you want it before or after it was processed? There are a lot of cleaning, categorizing, feature extraction steps. The data typically undergoes a lot of analysis, extra annotation, bucketing, and transformation.

If the pre-train was done in stages, and the training process was complicated, how much hand-holding do you need to replicate that process?

Do you need all of the scripts to assist with these processes? All of the infra and MLOps pieces? There's a lot of infrastructure to just move the data around and poke it.

Where are you going to host those terabytes or petabytes of data? Who is going to download it? How often? Do you expect it to be downloaded as frequently as the Linux kernel sources?

Did you scrub it of PII? Are you sure?

And to clarify, we're not even talking about trained models at this point.

We use pop-cultural references to communicate all the time these days. Those don't necessarily come from only the most commonly known sections of these works, so the AI would necessarily need the full work (or a functional transformation of the work) to be able to hit the theoretical maximum of the ability to decode about and reason using such references. To exclude copyrighted works from the training set is to expect it to decode from the outside what amounts to humanity's own in-group jokes.

That's my formal argument. The less formal one is that copyright protection is something that smaller artists deserve more than rich conglomerates, and even then, durations shouldn't be "eternity and a day". A huge chunk of what is being "stolen" should be in the commons anyway.

I truthfully cannot think of a single model that satisfies your criteria.

And if we wait for the the internet to be wholly eaten by AI, if we accept perfect as the enemy of good, then we'll have nothing left to cling to.

> And the question is also pretty clear: did $company steal other peoples work?

Who the hell cares? By the time this is settled - and I'd argue you won't get a definitive agreement - the internet will be won by the hyperscalers.

Accept corporate gifts of AI, and keep pushing them forward. Commoditize. Let there be no moat.

There will be infinite synthetic data available to us in the future anyway. And none of this bickering will have even mattered.

Fireship consistently makes some of the most entertaining tech content out there

> But those benchmarks are in general fairly narrow. They don't really measure the "broader" intelligence we are after.

I think a general model that can

- finish nethack, doom, zelda and civilization,

- solve the hardest codeforces/atcoder problems,

- formally prove putnam solution with high probability, not given the answer

- write a PR to close a random issue on github

is likely to have some broader intelligence. I may be mistaken, since there were tasks in the past that appeared to be unsolvable without human-level intelligence, but in fact weren't.

I agree that such benchmarks are limited to either environment with well-defined feedback and rules (games) or easily verifiable ones (code/math), but I wouldn't say it's super narrow, and there are no non-LLM models to perform significantly better on these (except some games); though specialized LLMs work better. Finding other examples, I think, is one of the important problems in AI metrology.

> So, yes, we are kind of left with vibe checks, but in theory, we could do more; take a bunch of models, double-blind, and have a big enough, representative group of human evaluators score them against each other on meaningful subjects.

You've invented an arena (who just raised quite a lot of money). Can argue about "representative," of course. However, I think the SNR in the arena is not too high now; it turns out that the average arena user is quite biased, the most of their queries are trivial for LLMs, and for non-trivial ones, they cannot necessarily figure out which answer is better. MathArena goes in opposite directions: narrow domain, but expert evaluation. You could imagine a bunch of small arenas, each with its own domain experts. I think it may happen eventually if money flow into AI continues.

None of those players you mention actually need to create a buzz. People will do it for them for free. DeepSeek joined this group after releasing R1.

Despite constant protestations of hype among the tech crowd, GenAI really is big enough of a deal that new developments don't need to be pushed onto market; people are voluntarily seeking them out.

Every time someone asks me what hardware to buy to run these at home I show them how many thousands of hours at vast.ai you could get for the same cost.

I don't even know how these Vast servers make money because there is no way you can ever pay off your hardware from the pennies you're getting.

Yeah, to run the full precision model you need either two 8xH100 nodes connected via Infiniband or one 8xH200 node or one 8xB200 node.

Not for the GPU poor, to be sure.

It is enough to run the dynamically quantised 1.56 bit version I believe, which is fun to play around with.

Flash is better than Pro in coding? Whoa... [makes a note to try a few things later this day]

Out of curiosity, how did you gauge that?

It's "manipulating the market" only when your geopolitical adversary brings the competition.

Hard to say exactly how it will affect the market, but IIRC when deepseek was first released Nvidia stock took a big hit as people realized that you could develop high performing LLMs without access to Nvidia hardware.

Took me until this thread to remember that in the 90s we had "freeware".

The weights are the source. It isn't as though something was compiled into weights. They're trained directly. But I know what you mean, it would be more open to have the training pipeline and souce dataset available.

You can fine-tune their weights and release your own take.

E.g. see all the specialized third-party models out there based on Qwen.

"Open-source" is the wrong word here, what they mean is "you can modify and redistribute these weights".

There is work to try to reproduce (the original) R1: https://huggingface.co/open-r1

I won't call it "binary blob". Safetensors is just a simple format for storing tensors safely: https://huggingface.co/docs/safetensors/index

That can, today, collect that data and sell it. There is work being done to add TEE, but it isn't live yet.

Every prompt is valuable.

They can't host DeepSeek because it's too big. Their chips have 230mb of memory, so it will take them ~3000 chips to host the model + (possible large) number of chips to keep kv cache. I bet it's just too hard to bring such topology online at all, and impossible to make even near to be profitable.

Huh. I heard a podcast with the founder talking about their custom hardware, but quantization would explain it.

Why repeat this nonsense when it’s so trivial to just check. The reason Groq is fast is because they employ absolutely ludicrous amounts of SRAM. (Which is 10 times faster than the fastest VRAM.)

they responded to my tweet last year and said they didn't quantize the models.

> Some of it I can even regurgitate almost exactly

If you (or any human) violate copyright law, legal redress can be sought. The amount of damage you can do is limited because there's only one of you vs the marginal cost of duplicating AI instances.

There are many other differences between humans and AI in terms of capabilities and motivations to f the legal persons making decisions.

:-) I like the symmetry of this. If I want to keep my creations outside the hands of others, I can keep them private. I don’t have to publish these words or broadcast them to the world. I could write this on my laptop, save it in a file, and keep it to myself. Fine.

However, once these words are broadcast—once they’re read, and the ideas expressed here enter someone else’s mind—I believe it’s only fair that the person on the receiving end has the right to use, replicate, or create something from them. After all, they lent me their brain—ideas that originated in my mind now live in theirs.

This uses up their mental "meat space," their blood sugar, and their oxygen—resources they provide. So, they have rights too: the right to do as they please with those ideas, including creating any and all data derived from them. Denying them that right feels churlish, as if it isn’t the most natural thing in the world.

(Before people jump on me:- Yes, creators need to be compensated—they deserve to make a living from their work. But this doesn’t extend to their grandchildren. Copyright laws should incentivize creation, not provide luxury for the descendants of the original creator a century later.)

This is a fundamental misunderstanding of copyright.

Copyright isn’t violated when someone consumes a copyrighted work.

Copyright is violated when a copyrighted work is used by someone who isn’t the author to generate profit without prior permission.

You can read a copyrighted book and remember it. You cannot copy it and sell copies. If you want to excerpt it you must give credit and there are limits to what’s considered “fair use”.

Even if training on the copyrighted material is OK, just providing a data dump of it almost certainly is not.

There is a "keep doing what you're doing, as we would want one of our companies to be on top of the AI race" signal from the governments. It could've been stopped, maybe, 5 years ago. But now we're way past it, so nobody cares about these sort of arguments.

Only goes to shown how strong their brand already is in the eyes of the public. I question how much active maintenance it takes, though. In my eyes, they already won big with ChatGPT - the name still is synonymous with LLMs to the general population. Everyone knows what ChatGPT is. Few known what Claude or Gemini is; arguably, more people know what Deepseek is thanks to the splash they made tanking NVidia stock and becoming part of general news coverage for a few days. Still, for regular folks (including business folks in tech industry, too), they all are, respectively, "that other ChatGPT", "ChatGPT from Google" and "that Chinese ChatGPT". It's a pretty sticky perception.

Yes, I know them too, I live there!

The hardware is great, CANN is not CUDA yet.

It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai

While true, computers in general alreay had the ability to perfectly replicate data, hence blank media tax: https://en.wikipedia.org/wiki/Private_copying_levy

I think the reason for all the current confusion is that we previously had two very distince groups of "mind" and "mindless"*, and that led to a lot of freedom for everyone to learn a completly different separation hyperplane between the categories, and AI is now far enough into the middle that for some of us it's on one side and for others of us it's on the other.

* and various other pairs that are no longer synonyms but they used to be; so also "person" vs. "thing", though currently only very few actually think of AI as person-like

Yes, but gun control and dual-use export regulations are both stupid. We need fewer tool-blaming laws, not more.

(Standing by for the inevitable even-goofier analogy comparing AI with privately-owned nuclear arsenals...)

> Finetuning isn't reverse engineering

Fully agree, it isn't. Reverse engineering isn't necessary for modifying compiled program behaviour, so comparing it to finetuning is not applicable. Finetuning applied to program domain would be more like adding plugins or patching in some compiled routines. Reverse-engineering applied to models would be like extracting source documents from weights.

> Finetuning is a standard supported workflow for these models.

Yes, so is adding mods for some games, just put your files in a designated folder and game automatically picks it up and does required modifications.

> Also, the "redistribute" part is key here.

It is not. Redistributability and being open source is orthogonal. You can have a source for a program and not be able to redistribute source or program, or you can redistribute a compiled program, but not have it's source (freeware).

Sure you can. It's often legally protected activity. You're just limited to distributing your modifications without the original work.

I'm assuming the Intelligence Revolution, multiplying Human Intelligence with machines, will have the same effect as the Industrial Revolution had, on multiplying human physical strength. That multiplied the GDP by a factor of ~20 times, hockey stick like, in a fairy short time, a century or two.

Aren't the datasets mostly shared in torrents? They probably won't bitrot for some time.

Obviously it's what we do with them, the biotech manufacturing and nuclear power production revolution happened pre AI. The reason it hasn't replaced petroleum is economic and social.

Why would you store the data after training?

If you try to reproduce various open datasets like fineweb by scraping the pages again, you can't, because a lot of the pages no longer exist. That's why you would prefer to store them instead of losing the content forever.

It's not "all of the text", it's like less than 100 trillion tokens, which means less than 400TB assuming you don't bother to run the token streams through a general purpose compression algorithm before storing them.

> Even the simplest operations like matrix multiplication will give you slightly different results depending on the hardware you're using

That has everything to do with implementation, and nothing to do with algorithm. There is an important difference.

Math is deterministic. The way [random chip] implements floating point operations may not be.

Lots of scientific software has the ability to use IEEE-754 floats for speed or to flip a switch for arbitrary precision calculations. The calculation being performed remains the same.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

That would fall under "Feed the same data in and you'll get the same weights out." Lots of deterministic algorithms use a random seed.

So is there no “introduce randomness” at some step afterwards? If not, I would guess these models would be getting stuck in a local maxima

Can you point to something that says it's not? The only source of non-determinism I've read of affecting LLM training is floating point error which is well understood and worked around easily enough.

> I have little doubt that some implementations aren't deterministic

Not some of them; ALL OF THEM. Engineering training pipelines for absolute determinism would be, quite frankly, extremely dumb, so no one does it. When you need millions of dollars worth of compute to train a non-toy model are you going to double or triple your cost just so that the process is deterministic, without actually making the end result perform any better?