Practically speaking, is it possible for NVIDIA to "pull the rug" later, intentionally or otherwise, by subtly changing the behaviour of this out-of-doc instruction on new architectures?

From this point of view I don't understand what's happening between the actual SOTA models practice and the academic models. The former at this point are all MoEs, starting with GPT4. But then the open models, if not for DeepSeek V3 and Mixtral, are always dense models.

Open AI™ (with a space)

Not really open source. For a truly open source model, check out OLMo 2 from AI2:

https://allenai.org/blog/olmo2

They literally share everything you need to recreate their model, including the data itself. This is what they say on that link above:

> Because fully open science requires more than just open weights, we are excited to share a new round of OLMo updates–including weights, data, code, recipes, intermediate checkpoints, and instruction–tuned models—with the broader language modeling community!

> The incentive behind the work of DeepSeek might very well be wrong (something along the lines of a state-sponsored attempt at shrinking the US first mover advantage in AI to nil)

In the space of international relations, right and wrong don't apply nearly as much. Is open sourcing this any more "wrong" than the export ban on high end Nvidia GPUs?

The open sourcing by DeepSeek (presumably with CCP consent) just happens to be good for both the CCP and the broader open source AI community at the same time, but don't take it as some kind of principled stance by them.

Finding ways to take away other countries' competitive advantages is a major activity off all governments, large and small.

Yes, there's some in the csrc/kernels directory. Search for 'asm' to find uses of it.

> the PTX that everyone was looking forward to

explanation for the rest of us why this is so important?

"For extreme performance, we discover and use a behavior-out-of-doc PTX instruction: ld.global.nc.L1::no_allocate.L2::256B. This instruction will lead to an undefined behavior: accessing volatile GPU memory with non-coherent read-only PTX modifiers .nc. But the correctness is tested to be guaranteed with .L1::no_allocate on Hopper architectures, and performance will be much better. If you find kernels not working on some other platforms, you may add DISABLE_AGGRESSIVE_PTX_INSTRS=1 to setup.py and disable this, or file an issue."

this might help: https://x.com/main_horse/status/1894215779521794058/photo/1

We're 2/3rds of the way there.

We need:

1. Open datasets for pretrains, including the tooling used to label and maintain

2. Open model, training, and inference code. Ideally with the research paper that guides the understanding of the approach and results. (Typically we have the latter, but I've seen some cases where that's omitted.)

3. Open pretrained foundation model weights, fine tunes, etc.

Open AI = Data + Code + Paper + Weights

Opening data is an invitation to lawsuits. That is why even the most die-hard open source enthusiasts are reluctant. It is also why people train a model and generate data with it, rather than sharing the original datasets.

These datasets are huge, and it's practically impossible to make sure they are clean of illegal or embarrassing stuff.

For understandable reasons

It is pirated material / material that breaks various terms of service but as I understand it is the stuff you can see in Anna's Archive and a bunch of "artificial" training data from queries to OpenAI ChatGPT and other LLMs.

Yes, releasing training source code code is like releasing the source code of a compiler used to compile and link the binary.

Lets say you took GCC, modified its sources, compiled your code with it and released your binaries along with modified GCC source code. And you are claiming that your software is open source. Well, it wouldn’t be.

Releasing training data is extremely hard, as licensing and redistribution rights for that data are difficult to tackle. And it is not clear, what exactly are the benefits in releasing it.

PyTorch had the "first thing that didn't suck" advantage and now it has a completely dominant marketshare that prevents better alternatives from emerging. Where it sucks (e.g. on macOS) there are popular alternatives. But it's hard to be enthusiastic about a DL framework in 2025 which does not have native high performance quantization support, for example. Or one where FSDP is crudely bolted onto the side. They say "usability above all else", but I consider such things to be major usability deficiencies, which need to be addressed. But because PyTorch does not have to fight for marketshare, it'll be years before we see anything usable there.