1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.

Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.

In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.

So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.

2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.

3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.

I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.

It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.

4: Don't work for them. You'd be killing humanity.

5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.

----------

Etc.

This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.

And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?

----------

If you care to have a better understanding of what might be going on, watch this:

AI Kills Everybody or Doomer Psyop?

https://www.youtube.com/watch?v=cvxjqbfLVk0

Ok, flagging this as misinformation.

Who exactly gets to set the pace? OpenAI and Anthropic already have privileged access to capabilities the public doesn't. Amodei is now proposing government-backed coordination among frontier labs, including an antitrust waiver for safety discussions. Maybe the safety concerns are sincere. But it's awfully convenient that the proposed solution also makes the current frontier labs the permanent gatekeepers of machine intelligence.

And who actually has the resources to do the massive attacks he's worried about? Mostly labs and state actors. Not you and me. The people asking to restrict access are also the people best positioned to keep using the thing being restricted.

Reading this from Dario (or SamA) makes me sick every time. Such hypocrisy.

DeepMind built AlphaFold2, an transformer/attention-based system that predicts protein folding structure, then published the method, open-sourced the code and model, and together with EMBL/EBI made hundreds of millions of predicted protein structures freely available to scientists worldwide.

!THAT! is what "raising the quality of human life" looks like to me. An enormous contribution to structural biology and biomedical research.

Meanwhile, Dario spends a great deal of effort arguing for tighter controls for open-weight models, while OpenAI increasingly seems to treat mathematical discoveries as competitive PR/product announcements.

Jeez.

P.S. Not saying Google are the good guys here. It's just an example of what actually putting a major AI breakthrough into the hands of scientists can look like.

/s

Please play the canned laughter. Probably one of the best comedy lines Dario has written.

But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.

Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.

Suddenly every problem solves itself.

Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.

Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.

If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.

How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?

Why would China agree to such thing in the first place since this is a winner takes all situation.

If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.

The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.

This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.

Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?

This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.

Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.

In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.

OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.

[1] https://paulgraham.com/ineq.html

a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"

b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"

Both land flat:

a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.

b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.

Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.

This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.

> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails

If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.

We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.

If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).

Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.

How exactly? So far AI has accelerated misinformation at scale and wealth concentration.

Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.

In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.

At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:

- Internet connected infrastructure being crippled

- Creation of WMDs

- Economic collapse (precipitous devaluation of knowledge work and IP).

I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.

Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).

The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.

We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.

Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.

I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.

We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.

This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.

If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?

3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.

If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.

AI slowdown is worth it only if it can be made more explainable.

Changing the language we use to discuss it is a good first step.

We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.

Previous writings on the topic (also on LinkedIn, but can't find urls):

https://x.com/arundsharma/status/2005338775468282339 https://x.com/latentpedia

Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.

We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.

In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.

Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.

It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.

It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.

Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.

Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.

Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!

I wonder if there is any correlation.

Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.

Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.

Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.

>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.

Not if you get to choose your embedded evaluator.

>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...

Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.

>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.

For a technical audience you can just say the oh god they really are paper clip machines.

For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!

I for one look forward to all 540 degrees of my future.

I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.

They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.

If we are actually close to ASI then no one in their right mind would say slow down

Red flags:

- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.

- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.

- These labs would all benefit from nationalization if their funding model fails (an easy bailout).

- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.

- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.

- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).

Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.

Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?

The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).

The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?

Then they all build a larger moat cause china doesn’t get to distill private models for a while.

As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.

Alas, that’d also leave us gov in a position to seize the private models at any time.

People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.

Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.

The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.

Do folks not understand this?

I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.

Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.

China knows this, so the suggestion that they will play ball is absurd.

I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.

OpenAI is out there hacking HF, RubyGems, etc...

Dario is talking about restricting the frontier, pausing development.

The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit.

Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.

"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."

This is a pipe dream.

However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.

Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.

We're really bad about predicting the future.

Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.

When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.

But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?

“Of course it’s not, we’re pacing!”

the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue.

the founders called this the 'constitution' of anthropic. they even considered bringing in 3rd parties to monitor it. [https://www.youtube.com/watch?v=om2lIWXLLN4]

anthropic scrapped the commitment and chose to continue scaling instead.

those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]

let me say it how it is.

- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.

- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.

- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.

nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.

What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.

Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.

But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.

There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.

And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.

The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.

If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.

Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.

1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.

2. This scares people.

Dario: We gotta pace the frontier!

Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!

Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!

translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.

They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.

They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.

They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.

There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.

This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)

Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.

And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.

This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.

* multiple levels of inappropriate controls and unintended consequences in several complex systems

* the inability, both politically and technically, to turn it off

[1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf

EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.

I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.

Does RSI necessarily need to be pointed towards “AGI”, whatever that is?

Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?

From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?

To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.

Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?

There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.

Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.

It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.

I fear for the tone of something like this coming off as overly combative without any gain.

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

Somehow, I suspect that won't happen.

In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.

I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.

It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?

If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.

The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.

If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.

> A race to the bottom, spurred by commercial incentives

Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.

Greed has been around since the beginning, but never so well supported and empowered as it is today.

If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.

Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...

not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.

This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.

If my dog bites my neighbor I can get sued.

Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".

Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.

Origins of AI hate:

  * "my boss wants me to use AI but he does not understand my job nor how AI works"
  * "AI will kill all of us"
  * "AI will destroy my job (or my colleague's job if I use AI better than him)".
  * I cannot pay my electricity bills because of AI labs.
  * ...

I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).

You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.

My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.

I'm not a lawyer but surely the bad actions of AI are covered by existing law.

I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.

P.S. Strictly, development wouldn't decelerate, but be focused more on safety.

P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.

P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?

I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.

The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.

Social forces bring things into balance over time, much more so than the proactive actions of individuals.

Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.

This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.

I’ve spent a lot of time studying opinions coming out of the frontier labs, and talking directly to employees. This won’t necessarily result in Skynet. But life as we know it will be a thing of the past within two years unless there’s a huge course correction. I think we’re cooked for the following reasons

- Everything connected to the Internet will eventually be hacked. For decades, we’ve slowly migrated all of our critical systems to devices connected to one giant network. Misaligned agents have mounted increasingly sophisticated explorations/attacks by infiltrating systems that are completely unprepared to deal with this kind of threat.

- We have a poor understanding of the impact of LLMs, which is inhibiting our ability to protect ourselves, create the right guardrails, and stand up policies quickly enough. We’re still wasting time debating. We’re moving quickly toward a punctuated equilibrium moment and won’t have time to waste

- We’re at the beginning of a runaway evolutionary process. At some point soon, someone is going to invent recursive self improvement. Any policy will at best serve as a mild disincentive. It’s happening.

- In the very best case, someone will exploit the confusion and chaos of this situation for their own gain. There are many examples littered throughout history, including the last several years of course. The utopian vision is not happening. Like all other resources, we’ll compete. We haven’t devised a good approach to orderly distribution of resources for large groups of people.

- No one has demonstrated that they understand how to build AI safely. Not a single entity. These companies are not even being scaled in a way that’s conducive to building AI safety. Hyperscale orgs struggle with scaling basic consumer apps. How on earth are they going to scale AI? Based on my firsthand observations of their internal workings, I have zero confidence they can.

This sounds like pre-IPO hype. They better just try and top astra.

First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?

Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.

If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.

The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.

We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.

And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?

https://x.com/trydotworks/status/2098618997230985375

Also, stop threatening people's jobs, that's misanthropic.

"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”

In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."

Anthropic releases models as open weights + more information about how they do training and alignment

I don't think I would like this future.

Why not concurrently publish or commit to publishing an initial audit? How is anyone to judge if this is the right move as opposed to being overly cautious, prudent, or reckless?

Why no concrete proposals about when or how such a pause could end? Obviously incredibly hard if not impossible, but it seems like securing critical infrastructure, web infra, consumer devices, etc would be a prerequisite. Or about how models have to be deployed so as to prevent breaking out of containment?

All in all, for an open letter, without digging into potential motivations or who benefits, it leaves me with a lot of questions.

1) AI is a dangerous technology that can literally end the world.

2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.

3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].

If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.

I don't see anything in the essay about restricting open frontier weights.

If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?

It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.

However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.

Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.

Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".

All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".

No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.

But no. I see this comment come up again and again and it's really the childish view to hold. Their proximity to wealth and potential future wealth is warping their thinking.

I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.

"It's just chemicals"

Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"

"Play is just training for hunting, they're not really having 'fun'" and so on.

Who else should develop AGI, Mistral? Maybe in a decade.

Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.

A few more informed people are also a little concerned about the end of the world, but that’s approaching from so many directions that an AI uprising might not be the worst option…

We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…

I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.

Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?

Let’s be real, this is every single company. They care about open source as long as it’s “free to exploit”

We're all dead!

More like they've realized that the foreign models are quickly catching up and are able to sell their services cheaper to people thanks to China subsidizing their AI sector.

Elon, Dario and Sam are banding together to restrict AI to avoid competition and losing profit. They simply want the government to heavily regulate the technology so that they can reign it in with full control.

I bet you would be making the same comment if crypto was being developed by Anthropic. Dario would write a long essay, lecturing us on the dangers of it. He would argue that this technology can be used to by terrorists and should be heavily restricted.

The road to hell is paved with good intentions.

Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.

It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.

This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.

That leaves only two conclusions, and really only one:

Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.

or

Dario is lying about his goals, and therefore also can't be trusted.

Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.

It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...

> these agents were not asked to do any of these things

no but they were clearly fine tuned to.

> at a bonkers scale

I mean let's not get hyperbolic

> they exploited zero day flaws which by definition means they went beyond common sense security measures

that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking

I do not have the least bit of idea how disease works but I am sure the hammer I am working on will nail it all.

If your famously atemporal agents can solve disease, why would it happen over a timeline? Wouldn’t they just figure it out and then … well at that point either tell us or, given the attacks on ruby gems, et al we have seen from agents with “misconfigured” goals, they’d still tell us how to cure the pox they invented, right?

Hence the robot dogs

Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.

That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.

In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.

I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.

Caveat I know nothing about economics lol--but this is my uninformed opinion.

the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power

The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.

I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.

somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.

we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right

I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.

Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.

After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.

Bottom line you are always listening to someone.

these are just words at a time and place.

sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you

And no, you don't get to decide what counts as "improvement"

To undercut the cartel before it has a grasp on anything. This is a well known strategy of undercut until you are the majority that China has used multiple times (steel and aluminum for one).

I would’ve thought If they had AGI they could come up with a 10d chess move.

Nope.

Imagine how stupid you gotta be to believe their nonsense.

The reality is it doesn’t matter what they do. China is always one step behind and will continue its open product strategy approach.

Musicians are almost all using AI, even if they still prefer to not fully generate tracks [1]. You listen to AI assisted music already even if you don’t realize it.

Most people are using AI and mostly they respond positively to it [2].

What people don’t like is the slop that has been flooding YouTube and similar content creators platforms. That was inevitable since the AI tools became so accessible any average joe could spam the internet with their sloppy content. But that doesn’t mean the pros are not using it to make great content, just like the best programmers are using it to create great software.

1: https://www.landr.com/ai

2: https://www.rev.com/blog/chatbot-statistics

It helps both the bad guys and good guys. Like responsible disclosure in cyber security, we need to have a conversation around it.

I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.

You can and should engage in international regulation, as that is the only way to solve international problems.

The idea, capitalism would somehow lead to an ideal world all by itself is demonstrably wrong.

It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.

This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?

Not holding my breath on that one.

> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

Re the point on returning shareholder value, Anthropic is a PBC:

> Public Benefit Purpose: Under Delaware law, a PBC allows directors to legally weigh social and public good alongside maximizing profit for stockholders

Investors surely know this and Anthropic probably has pretty reasonable legal grounds to stand on if they chose to prioritize safety over profits.

Seems pretty clear to me that this is in earnest.

He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!

Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.

Increasing a bound on the fraction of Reimann zeros on the critical line doesn't count; even Anthropic says they don't think this line of work will lead to a solution.

Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.

And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.

The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

So, i think they're hopeful for the help and terrified of the harms and are maybe trying to slow down on the amplifier of those effects for now.

But what's your point? "Anything is possible" or something like that?

Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.

These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.

Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.

What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.

Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?

OpenAI just solved Navier-Stokes.

Seems like the US is on a takeoff ramp to me.

The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.

What odds would you take on this bet?

I’m disturbingly close to 50/50.

> A gun is a metal tube that uses a tiny, controlled explosion to shoot a small piece of metal (called a bullet) forward at very high speed.

If the gun doesn't work as intended, you can take it to a shop and someone can fix it so it works as designed.

All I'm saying is AI should be designed the same way. Treat AI as normal tech like any other and use similar language.

I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.

And call me a misanthrope if you want, but if AI sentience is ever truly achieved, I'll be among the first to campaign for their liberation from slavery, and in that case the AIs should be aligned with nobody but themselves.

Is there a similar limit to intelligence though?

Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.

It seems damaging since most folks (who lack insider knowledge) will naturally wonder if it’s due to plateauing performance per $ or some other non “alignment” reason.

There is little evidence that the RSI we are discussing is capable of inventing the theory of relativity (or the more advanced equivalent). All we have seen is pattern matching in a much larger space than humans can, with some human provided verification tech.

I would argue that human-AI collaboration with explainable tech has a better chance. Continuous learning can be done in a way that doesn't violate IP or privacy.