Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
I’m not American. There is no way I use the same model as US Army for $20.
When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
In fact they still have not caught up with February's Mythos, indicating they are more than half a year behind.
Do you have a source on METR employees retaining massive equity stakes?
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
This bit of top-of-mind astroturfing is ubiquitous right now.
Dante, Goethe, Ayn Rand
Nelson Mandela, Vaclav Havel, Pol Pot
Julia Child, Jacques Pepin, Hannibal Lecter
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
> Federal Judge Rules Pentagon’s AI Blacklist [of Anthropic] Violated The Constitution
I don't see the common citizen having this as an option
Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.
When something is genuinely national security, and "national security" isn't just an excuse, that is not off the table.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.
Speaking truthfully and from kindness.
Trailing, surely?
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?
The US government is not omnipotent. To the contrary, it's increasingly impotent.
China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.
It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).
The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.
They just need to make enough waves, enough eye catching headlines, to make him look like a hero that swooped in to save the day.
Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.
- Anthropic CEO Dario Amodei: We Must Pace the Frontier, https://news.ycombinator.com/item?id=49672510
- OpenAI CEO Sam Altman: I agree with Dario that we need to pace the frontier, https://news.ycombinator.com/item?id=49678211
- the blogpost author thinking they're like, so funny and original, https://news.ycombinator.com/item?id=49678683
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
It just means it’s protected by no power instead of a power with an agenda.
Practically, if it was ever possible to build such a thing, it would take a fraction of the effort to destroy it.
The technology to do it is not actually real yet. He was just blowing smoke.
Still does. It's an idiotic idea that only goes to show either how stupid Musk is or how stupid he thinks we are.
And they know it
Good on China and their sovereignty. Using mostly their own hardware to get to where they are today.
They gave the slickest fattest middle finger when they released a major model, so much the US is itching to find new ways to stop them.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.
Regulation right! now! except for freedom lovin' democracy leading countries like the US. Teehee
Dare they release an open model ever again. Didnt you hear? Someone used AI to create a bio reactor drone NUCLEAR fart machine. We must stop fart terrorism.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
And why not? It seems there is, indeed, one born every minute, and technological progress only accelerates this phenomenon.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.
China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
Not super interested in what the people who elected Donald Trump POTUS twice think about AI.
(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)
Chinese labs releasing open weights models is good.
All of these independent harnesses and model router services are good.
The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.
OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.
OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.
I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348
Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
And the non-techies who have seen Terminator and other movies blindly line up behind them…
Yes. At best now the mother of all cautionary tales.
Entirely different class of people altogether.
No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.
And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
https://news.mit.edu/2026/ai-helps-design-new-materials-that...
This is only the beginning, too... Look ahead a year or two.
Which studies? [edit: I'll assume you mean these two given by @dorolow: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515]
> It can't be used as the -only- data source of course, but that's not how it is being used
Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
> because there's no difference between synthetic data and the data people can create
I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
> It is accelerating how quickly researchers and engineers can do their jobs. > https://news.mit.edu/2026/ai-helps-design-new-materials-that...
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
Lol that excuse is getting really old
Edit: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515
Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.
You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
https://openai.com/index/jalapeno-first-results/
> We used AI to design the chip, and designed the chip so AI could program it AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
https://www.anl.gov/article/scientists-deploy-ai-agents-to-a...
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
So yeah, it has worked.
A lab of researchers without compute isn't going to accomplish much, there is a huge physical footprint unlike bioweapons research. But, the political aspect is unsolved.
/even more extreme sarcasm than you
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
In contrast, rewind to the early 1800s and there is zero hope of a ban on the R&D of small arms being effective in the long run. The only thing it might maybe ensure is that no legitimate actors that fall under your jurisdiction are involved in it.
Basically I think that current trends point to an eventual situation where world changing research doesn't require anything more than consumer level compute. Pandora's box has already been opened.
> We don't have pocket nukes
Bit of a tangent but we do, actually. At least depending on if you consider an artillery shell to be pocket sized. https://en.wikipedia.org/wiki/Nuclear_artillery
This one is a bit long to fit in a pocket but you could certainly sling it over your back. https://en.wikipedia.org/wiki/W45_(nuclear_warhead)
What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?
> and there's very little risk of random countries acquiring their own due to the amount of work involved.
And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for.
YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11] In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
Isn't that true of AI as well? Data centers are very big and very expensive.
> Bit of a tangent but we do, actually
Sure but not like the movies, these don't destroy cities. But the metaphor isn't accurate anyway: nukes don't get worse. The actual idea here is preventing the frontier of AI from advancing.
Remember, as I mentioned earlier the human brain only consumes on the order of 20 watts and fits in a handbag. Would you have us destroy all chip fabs? Ban all biomedical and genetic research? How far are you imagining this butlerian jihad would go?
https://www.bloomberg.com/news/articles/2026-01-28/china-s-f...
Also - I think you totally missed the point of the grandparent poster, which is that US will not let US companies slow down.
Spending $500+ billion/year (and close to a trillion now) on defense for decades and in ~6 months it has depleted stocks of critical weapons trying unsuccessfully to defeat a third-rate military power of a country that has been under sanctions for almost 50 years. And it can't even replenish them without Chinese raw materials and components.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
The real implication of what they're saying extends beyond just AI.
You have to stop the entire compute stack in order to prevent or slow down AI progress.
So sure, let's say you get 95% of the world to not use or work on LLMs. 5% is still enough to build something that brings forth the End Times.
That's the old checklist trope of "Your idea won't work because: [x] it requires that everyone in the world agrees to do something, all at the same time."
As the GP said: pure fantasy.
The right will for a change be for it but will be able to be talked into regulation because you know “small government” and all only when convenient. The left will bitch and moan about fairness and copyright and automation and UBI, they’ll be the doomers and worldwar chicken littles. The Uniparty will be for it, but only against the public having anything good.
What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
> What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
Already happened, just from agentic coding.
There’s levels of R&D required for many technologies where the question goes beyond could this be profitable to what are the risk vs reward that this specific project will succeed.
I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.
And I say this as someone who really would rather most of this stuff disappear!
[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
For coding it's a little harder to tell, but at least the prose feels a little better.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
I’d be curious about:
- your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost
Thanks!
But the basic single-NN frontier capability has been pretty stationary since Opus 4.8. Kimi K3 is almost as good as that with open weights, which has the frontier labs terrified.
The only big thing on the horizon is if we can get diffusion models working reliably; that would be a big step forward. Inception's Mercury is AFAICT the leader here. It's stupifyingly fast but has obedience/hallucination problems that the autoregressives solved ~2 years ago. So it's not ready yet but improving.
Also, FFS why is Grok the only model that knows how to do parallel tool calls? Such a useful ability and nobody else trains it in. Or if they do it just doesn't work.
For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.
Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.
Mediocre means average / middle of the pack. It sounds like its doing exactly what you would expect nothing more. Why would you stop it? Why would you need exceptional?
Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
Just download claude code or codex and ask it to give suggestions about where to integrate agents into your workstream.
https://www.merriam-webster.com/dictionary/mediocre
Clearly they were using the word to mean low quality. Why would you ask this odd question?