Anthropic drops flagship safety pledge(time.com) |
Anthropic drops flagship safety pledge(time.com) |
The narrative on social media, this site included, is to portray the closed western labs as the bad guys and the less capable labs releasing their distilled open weight models to the world as the good guys.
Right now a kid can go download an Abliterated version of a capable open weight model and they can go wild with it.
But let's worry about what the US DoD is doing or what the western AI companies absolutely dominating the market are doing because that's what drives engagement and clicks.
They want Anthropic to enabling mass surveillance and autonomous attack systems with no human in the loop.
Hardly compares to a kid downloading a model to experiment with.
Is the reason to ban or block free open weight models that you're worried what kids will do with them?
I'd imagine the economic case to be made is that the Western AI companies will ultimately not be able to compete with free open weight models. Additionally, open weight models will help to spread the economic gains by not letting a few monopolies capture them behind regulatory red tape.
Finally, I'd say the geopolitics angle of why open weight models are better is that if the West controls the open source software that will power it will be able to reap the benefits that soft power brings with it.
making promises in good times is a real minefield hah
And it will be, as Warren Buffet puts it, a "Only when the tide goes out do you discover who's been swimming naked." moment.
Seems like it, yes.
I really miss the nerd profile who cared a lot more about tech and science, and a lot less about signaling their righteousness.
How did we get so religious/narcissistic so quickly and as a whole?
The entire playing field is kinda dissapointing, left or right. Which do you wanna be, self-righteous preening snob or batshit macho man?
I'm going for a blend, myself
We built a behemoth that rewards attention whoring and anti social behavior with money.
ok lol what a coincidence.
but setting aside the conspiracy. the article actually spells out the real reason pretty directly: Anthropic hoped their original safety policy would spark a "race to the top" across the industry. it didn't. everyone else just ignored it and kept moving. at some point holding the line unilaterally just means you're losing ground for nothing.
Even if it were ever done with good intentions, it is an open invitation for benefit hoarding and margin fixing.
Do you realy want to create this future where only a select few anointed companies and some governments have access to super advanced intelligent systems, where the rest of the planet is subjected to and your own ai access is limited to benign basal add pushing propaganda spewing chatbots as you bingewatch the latest "aw my ballz"?
Netflix said that they'd never have live TV, or buy a traditional studio, or include ads in their content. Then they did all three.
All companies use principled promises to gain momentum, then drop those principles when the money shows up.
As Groucho Marx used to say: these are my principles, if you don't like them, I have others.
Dark times and darker forests.
That doesn't even make sense.
What stops one model from spouting wrongthink and suicide HOWTOs might not work for a different model, and fine-tuning things away uses the base model as a starting point.
You don't know the thing's failure modes until you've characterized it, and for LLMs the way you do that is by first training it and then exercising it.
"We promise are not going to do __, except if our customers ask us to do, then we absolutely will".
What is the point? Company makes a statement public, so what?
Not the first time this company puts some words in the wind, see Claude Constitution. It's almost like this company is built, from ground up, upon bullshit and slop
Or least it didn't until the current regime.
The US does have autonomous defensive systems.
I could be wrong though, can you post your evidence? The closest I could find is loitering munitions.
Even so, a company shouldn't be forced to go against its ethics if those ethics help humans.
People are conscripted, they put on the uniform and become legitimate targets? It might as well be a robot doing the shooting. Same difference.
The pilot becomes responsible for those outcomes. For example indiscriminately killing civilians for example is a war crime. Its easier to get an AI to commit war crimes than humans.
The US is not the only country in the world so the idea that humanity as a whole could somehow regulate this process seemed silly to me.
Even if you got the whole US tech community and the US government on board, there are 6.7bn other people in the world working in unrelated systems, enough of whom are very smart
What would safety applied to the leading 3 mean to you anyways ?
So the stated concern of the west coast tech bros that we're close to some misaligned AGI apocalypse would be slightly delayed, but in the grand scheme of things it would make no difference
What a gigantic, absolute, pieces of s...
Not because of what they did, which is classic startup playbook but because of the cynicism involved, particularly after all the fuzz they've been making for years about safety. The company itself was founded, allegedly, due to pursuing that as a mission as opposed to OpenAI.
"Hi all, that was a lie, we never really cared." They only missed the "dumb f***s" remark, a la Facebook.
I have not read “If Anybody Builds It, Everybody Dies” but I believe that's also its premise.
Current GenAI is extremely capable but also very weird. For instance, it is extremely smart in some areas but makes extremely elementary mistakes in others (cf the Jagged Frontier.) Research from Anthropic and OpenAI gives us surprising glimpses into what might be happening internally, and how it does not necessarily correspond to the results it produces, and all kinds of non-obvious, striking things happening behind the scenes.
Like models producing different reasoning tokens from what they are really reasoning about internally!
Or models being able to subliminally influence derivative models through opaque number sequences in training data!
Or models "flipping the evil bit" when forced to produce insecure code and going full Hitler / SkyNet!
Or the converse, where models produced insecure code if the prompt includes concepts it considers "evil" -- something that was actually caught in the wild!
We are still very far from being able to truly understand these things. They behaves like us, but don't necessarily “think” like us.
And now we’ve given them direct access to tools that can affect the real world.
Maybe we am play god: https://dresdencodak.com/2009/09/22/caveman-science-fiction/
I think the Dario of today is very different to the Dario 3 years ago.
You are just one new feature announcement from Anthropic/OpenAI away from irrelevance.
Same as it was when people built their busineses on top of AWS a decade ago
https://apnews.com/article/anthropic-hegseth-ai-pentagon-mil...
1. Extremely granular ways to let user control network and disk access to apps (great if resource access can also be changed)
2. Make it easier for apps as well to work with these
3. I would be interested in knowing how adding a layer before CLI/web even gets the query OS/browser can intercept it and could there be a possibility of preventing harm before hand or at least warning or logging for say someone who overviews those queries later?
And most importantly — all these via an excellent GUI with clear demarcations and settings and we’ll documented (Apple might struggle with documentation; so LLMs might help them there)
My point is — why the hell are we waiting for these companies to be good folks? Why not push them behind a safety layer?
I mean CLI asks .. can I access this folder? Run this program? Download this? But they can just do that if they want! Make them ask those questions like apps asks on phones for location, mic, camera access.
Basicaly an EDR
Are people really attempting to have LLMs replace vision models in robots, and trying to agentically make a robot work with an LLM?? This seems really silly to me, but perhaps I am mistaken.
The only other thing I could think of is real-time translation during special ops with parabolic microphones and AR goggles...
It's just systems plumbing (surveillance) and AI. It's a combination of weaker technologies and consolidation of power.
This does not require a physical robot super AGI(though I would not be surprised if fully autonomous robots are not on the table already)
On the other hand, those organizations are operating in the best interest of Americans and the world right?
Surely, those agencies aren't just a trick of the rich people? Right?
The largest predictor of behavior within a company and of that companies products in the long run is funding sources and income streams, which is conveniently left out in their "constitution". Mostly a waste of effort on their part.
It isn't about the right answers, rather the expected answers.
The intention to start these pledge and conflict with DOW might be sincere, but I don’t expect it to last long, especially the company is going public very soon.
You can be correct and not play into their game by ignoring the name change completely.
It took Google probably 15 years to fully evil-ize. Anthropic ... two?
There is no "ethical capitalism" big tech company possible, esp once VC is involved, and especially with the current geopolitical circumstances.
The Amodeis' have just proven that the threat of even slight hardship will make them throw any and all principles away.
They’re pointless if they just get removed once you get close to hitting them.
And all the major corps seem to be doing this style of pr management. Speaks of some pretty weapons grade moral bankruptcy
The concept of "having a contract with society" doesn't even formally exist because companies would never sign one.
It's so much focus on implementation, and processes, and really really seems to consider the question of what even constitutes "misaligned" or "unethical" behavior to be more or less straight forward, uncontroversial, and basically universally agreed upon?
Let's be clear: Humans are not aligned. In fact, humans have not come to a common agreement of what it means to be aligned. Look around, the same actions are considered virtuous by some and villainous by others. Before we get to whether or not I trust Anthropic to stick to their self-imposed processes, I'd like to have a general idea of what their values even are. Perhaps they've made something they see as super ethical that I find completely unethical. Who knows. The most concrete stances they take in their "Constitution" are still laughably ambiguous. For example, they say that Claude takes into account how many people are affected if an action is potentially harmful. They also say that Claude values "Protection of vulnerable groups." These two statements trivially lead to completely opposing conclusions in our own population depending on whether one considers the "unborn" to be a "vulnerable group". Don't get caught up in whether you believe this or not, simply realize that this very simple question changes the meaning of these principles entirely. It is not sufficient to simply say "Claude is neutral on the issue of abortion." For starters, it is almost certainly not true. You can probably construct a question that is necessarily causally connected to the number of unborn children affected, and Claude's answer will reveal it's "hidden preference." What would true neutrality even mean here anyways? If I ask it for help driving my sister to a neighboring state should it interrogate me to see if I am trying to help her get to a state where abortion is legal? Again, notice that both helping me and refusing to help me could anger a not insignificant portion of the population.
This Pentagon thing has gotten everyone riled up recently, but I don't understand why people weren't up in arms the second they found out AIs were assisting congresspeople in writing bills. Not all questions of ethics are as straight forward as whether or not Claude should help the Pentagon bomb a country.
Consider the following when you think about more and more legislation being AI-assisted going forward, and then really ask yourself whether "AI alignment" was ever a thing:
1. What is Claude's stances on labor issues? Does it lean pro or anti-union? Is there an ethical issue with Claude helping a legislator craft legislation that weakens collective bargaining? Or, alternatively, is it ethical for Claude to help draft legislation that protects unions?
2. What is Claude's stance on climate change? Is it ethical for Claude to help craft legislation that weakens environmental regulations? What if weakening those regulations arguably creates millions of jobs?
3. What is Claude's stance on taxes? Is it ethical for Claude to help craft legislation that makes the tax system less progressive? If it helps you argue for a flat tax? How about more progressive? Where does Claude stand on California's infamous Prop 19? If this seems too in the weeds, then that would imply that whether or not the current generation can manage to own a home in the most populous state in the US is not an issue that "affects enough people." If that's the case, then what is?
4. Where does Claude land on the question of capitalism vs. socialism? Should healthcare be provided by the state? How about to undocumented immigrants? In fact, how does Claude feel about a path to amnesty, or just immigration in general?
Remember, the important thing here is not what you believe about the above questions, but rather the fact that Claude is participating in those arguments, and increasingly so. Many of these questions will impact far more people than overt military action. And this is for questions that we all at least generally agree have some ethical impact, even if we don't necessarily agree on what that impact may be. There is another class of questions where we don't realize the ethical implications until much later. Knowing what we know now, if Claude had existed 20 years ago, should it have helped code up social networks? How about social games? A large portion of the population has seemingly reached the conclusion that this is such an important ethical question that it merits one of the largest regulation increases the internet has ever seen in order to prevent children from using social media altogether. If Claude had assisted in the creation of those services, would we judge it as having failed its mission in retrospect? Or would that have been too harsh and unfair a conclusion? But what's the alternative, saying it's OK if the AI's destroy society... as long as if it's only on accident?
What use is a super intelligence if it's ultimately as bad at predicting unintended negative consequences as we are?
Also, agree with everything you say here. GIGO.
They also have never had any guarantees they wouldn't f*ck around with non-US citizens, for surveillance and "security", because like most US tech companies they consider us to be second/lower class human beings of no relevance, even when we pay them money.
At least Google, in its early days, attempted a modest and naive "internationalism" and tried to keep their hands clean (in the early days) of US foreign policy things... inheriting a kind of naive 1990s techno-libertarian ethos (which they threw away during the time I worked there, anyways). I mean, they only kinda did, but whatever.
Anthropic has been high on its own supply since its founding, just like OpenAI. And just as hypocritical.
Department of Defense is the official name, and they did have a choice: they could have stopped working with the military. But they chose money and evil.
It doesn't matter that there exists another name on some paper, when all official, ceremonial and public communications use this name. The old name is about as worthless as the constitution or the senate at this point. The executive branch has successfully taken over the country.
It's just a silly woke secretary choosing their own imaginary pronouns.
https://xcancel.com/elonmusk/status/2026181748175024510
I don't know where xAI got its training material from, but seeing Musk rewteeting that is refreshing.
The AI startup has refused to remove safeguards that would prevent its technology from being used to target weapons autonomously and conduct U.S. domestic surveillance.
Pentagon officials have argued the government should only be required to comply with U.S. law. During the meeting, Hegseth delivered an ultimatum to Anthropic: get on board or the government would take drastic action, people familiar with the matter said.
https://www.staradvertiser.com/2026/02/24/breaking-news/anth...
So... all acts are selfish because if it looks unselfish, that just means it was selfish in a hidden way?
I kind of wish they had forced the governments hand and made them do it. Just to show the public how much interference is going on.
They say it wasn't related. Like every thing that has happened across tech/media, the company is forced to do something, then issues statement about 'how it wasn't related to the obvious thing the government just did'.
Makes perfect sense!!
If a company is deemed a "supply chain risk" it makes perfect sense to compel it to work with the military, assuming the latter will compel them to fix the issues that make them such a risk.
Or, more likely, adding the "core safety promise" was just them playing hard to the government to get a better deal, and the government showed them they can play the same game.
* AI and states cannot peacefully coexist, and AI is not going to be stopped. Therefore, we must begin to deprecate states.
I think it's very unlikely that this is unrelated to the pressure from the US administration, as the anonymous-but-obvious-anthropic-spokesperson asserts.
We're at a point now where the nation states are all totally separate creatures from their constituencies, and the largest three of them are basically psychotic and obsessed with antagonizing one another.
In order to have a peaceful AI age, we need _much_ smaller batches of power in the world. The need for states that claim dominion over whole continents is now behind us; we have all the tools we need to communicate and coordinate over long distances without them.
Please, I pray for a gentle, peaceful anarchism to emerge within the technocratic leagues, and for the elder statesmen of the legacy states to see the writing on the wall and agree to retire with tranquility and dignity.
Humans are, by nature, forgetful and argumentative. Fourteen hundred years ago, the Qur'an said this unequivocally (20:115, 18:54, 22:8, 18:73). Not to moralize here, I'm just saying if camel-herders could build a medieval superpower out of nothing, they knew something we don't.
Any state or system that insists good humans are always nice, smart, cogent, and/or aware is doomed to fail. A Washington or a Cincinnatus that can get out of his own way (and that of society) is rare indeed, a one-in-a-billion soul. We shouldn't sit around and wait for that, while your run-of-the-mill dictator in a funny hat (or a funny toupée for that one orange fellow) has his way with us.
1. AI is military/surveillance technology in essence, like many other information technologies,
2. Any guarantee given by AI companies is void since it can be changed in a day,
3. Tech companies have no real control over how their technology will be used,
4. AI companies may seem over-valued with low profits if you think AI as a civil technology. But their investors probably see them as a part of defense (war) industry.
Given by anyone, actually.
Hegseth gives Anthropic until Friday to back down on AI safeguards
The safeguards dropped are when they will release a model or not based on safety.
The Friday deadline is to allow to use their products for mass surveillance and autonomous weapons systems without a human in the loop.
Anthropic hasn't backed down on those, yet. But they are in a bad situation either way.
If they don't back down, they lose US government contracts, the government gets to do what it wants anyway. It also puts them in a dangerous position with non-governmental bodies.
If they give into the demands, then it puts all AI companies at risk of the same thing.
Personally I think they should move to the EU. The recent EU laws align with Anthropics thinking.
Write essays about AI safety in the application.
An entire interview dedicated to pretending that you truly only care about AI safety and ethics and nothing else.
Every employee you talk to forced to pretend that the company is all about philanthropy, effective altruism and saving the world.
In reality it was a mid-level manager interviewing a mid-level engineer (me), both putting on a performance while knowing fully well that we'd do what the bosses told us to do.
And that is exactly what is happening now. The mission has been scrubbed, and the thousands of "ethical" engineers you hired are all silent now that real money is on the line.
The structural problem is that once you've taken billions in VC, safety becomes a negotiable constraint rather than a core value. The board's fiduciary duty runs toward returns, not toward whatever was in the mission statement. PBC status doesn't change that in practice — there's basically zero enforcement mechanism.
What's wild is how fast the cycle has compressed. Google took maybe 15 years to go from "don't be evil" to removing it from the code of conduct. OpenAI took about 5 years from nonprofit to capped-profit to whatever they are now. Anthropic is speedrunning it in under 3. At this rate the next AI startup will launch as a PBC and pivot before their Series B closes.
> The policy change is separate and unrelated to Anthropic’s discussions with the Pentagon, according to a source familiar with the matter.
It combines interpretation of meaning with ambiguity to allow the reporter to assert anything they want. The ambiguity is there to protect the identity of the source but it has to be a more discrete disclosure of information in return. If you can't check the person you can still check what they said.
I would be ok with direct quotes from an anonymous source. That removes the interpretation of meaning at least.
As it is written, it would not be inaccurate to say this if their source was the lesswrong post, or even an earlier thread here on HN.
Phrasing "A source with direct knowledge of the situation" might remove some of the leeway for editorialising, but without sharing what the source actually said, it opens the door to saying anything at all and declaring "That's what I thought they meant" when challenged.
It's unfalsifyible journalism.
Pledges are a cynical marketing strategy aimed at fomenting a base politics that works to prevent such a regulatory regime.
Is the implication here that Anthropic admits they already can't meet their own risk and safety guidelines? Why else would they have to stop training models?
https://www.npr.org/2026/02/25/nx-s1-5725354/nurses-emigrate...
Anthropic's market cap is going to be huge when they go public. Why do it on Nasdaq when there are so many other exchanges in the world?
I can't help but think about how Google once had "Don't be evil" as their motto.
But the thing with for-profit companies is that when push comes to shove, they will always serve the love of money. I'm just surprised that in an industry churning through trillions, their price is $200 million.
https://www.theverge.com/press-room/22772113/the-verge-on-ba...
On their podcast, they frequently bring up how tech company PR teams try to move as much conversation with journalists as possible into "on background", uncited, generic sourcing.
If they’re operating under a different definition of supply chain risk, I don’t have a clue.
It is not about disciplining them to get better.
1. So one option is about forcing them to produce something. You must build this for us.
2 The other option is saying they are compromised so stop using them all together. We will not use what you build for us at all because we don't trust it.
So . Contradictory.
How magnanimous! They are only thinking of others, you see. They are rejecting their safety pledge for you.
> “We didn't really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments … if competitors are blazing ahead.”
Oops, said the quiet part out loud that it’s all about money. “I mean, if all of our competitors are kicking puppies in the face, it doesn’t make sense for us to not do it too. Maybe we’ll also kick kittens while we’re at it”.
For all of you who thought Anthropic were “the good guys”, I hope this serves as a wake up call that they were always all the same. None of them care about you, they only care about winning.
But lucky for the AI companies, most of them are based in place that only has a government on paper and everyone forgot where that paper is.
https://apnews.com/article/anthropic-hegseth-ai-pentagon-mil...
But that government cannot trust the other government on the other side of the world to implement the same restrictions, so we find ourselves in this Nash equilibrium.
I mean, yes, that is actually how world works. That is why we need safety, environmental and other anti-fraud regulations. Because without them, competition makes it so that every successful company will fraud, hurt and harm. Those who wont will be taken over by those who do.
Now, you may notice that the government is also choosing not to regulate these companies...which is another matter altogether.
And soon enough, it won’t work at all because of it.
> Those who wont will be taken over by those who do.
And if you compromise on your core values because of money, they weren’t core values to begin with¹. “I want to be ethical but if I am I won’t get to be a billionaire” isn’t an excuse. We shouldn’t just shrug our shoulders at what we see as wrong because “everybody does it” or “that’s just business” or “that’s life”. Complacency and apologists are how a bad system remains bad.
https://www.newyorker.com/cartoon/a16995
¹ I’m willing to give leeway to individuals. You can believe stealing is wrong but if you’re desperate and steal a loaf of bread to feed your kid, there’s nuance. A VC-backed company is something entirely different.
To be fair, this is true in nearly all industries and for nearly all companies. Almost everyone is chasing money and monopoly. Not that it makes it right, just pointing out it isn’t unique or even interesting about the AI companies
It sounds like they are in a cutthroat market, and realised they couldn't afford to stake that principle. And that it wouldn't matter if they did – it would just assure them being handicapped in a field where no others followed suit.
Better hills to die on.
Was anyone fooled by this?
I mean, I know this is HN and there is a demographic here that gets all misty eyed about the benevolence of corporations.
It takes a special kind of naivety to believe in those claims.
Censorship?
> The policy change is separate and unrelated to Anthropic’s discussions with the Pentagon, according to a source familiar with the matter.
Their core argument is that if we have guardrails that others don't, they would be left behind in controlling the technology, and they are the "responsible ones." I honestly can't comprehend the timeline we are living in. Every frontier tech company is convinced that the tech they are working towards is as humanity-useful as a cure for cancer, and yet as dangerous as nuclear weapons.
We must build a moat to save humanity from AI.
Please regulate our open-source competitors for safety.
Actually, safety doesn't scale well for our Q3 revenue targets.
‘While there’s value in safety, we value the Pentagon’s dollars more’
That said, I'm not thrilled about this. I joined Anthropic with the impression that the responsible scaling policy was a binding pre-commitment for exactly this scenario: they wouldn't set aside building adequate safeguards for training and deployment, regardless of the pressures.
This pledge was one of many signals that Anthropic was the "least likely to do something horrible" of the big labs, and that's why I joined. Over time, the signal of those values has weakened; they've sacrified a lot to get and keep a seat at the table.
Principled decisions that risk their position at the frontier seem like they'll become even more common. I hope they're willing to risk losing their seat at the table to be guided by values.
The article yes, but we cannot be sure about its topic. We definitely cannot claim that they are unrelated. We don't know. It's possible that the two things have nothing to do with each other. It's also possible that they wanted to prevent worse requests and this was a preventive measure.
This cannot have been caused by that, unless they've also invented time travel.
I just hope something happens to USA before it can do damage to the world.
I don't know enough to evaluate this or other decisions. I'm just glad someone is trying to care, because the default in today's world is to aggressively reject the larger picture in favor of more more more. I don't know how effective Anthropic's attempts to maintain some level of responsibility can be, but they've at least convinced me that they're trying. In the same way that OpenAI, for example, have largely convinced me that they're not. (Neither of those evaluations is absolute; OpenAI could be much worse than it is.)
> The meeting between Hegseth and Amodei was confirmed by a defense official who was not authorized to comment publicly and spoke on condition of anonymity.
https://fortune.com/2026/02/24/hegseth-to-meet-with-anthropi...
"Defense Secretary Pete Hegseth has threatened Anthropic, saying officials could invoke powers that would allow the government to force the artificial intelligence firm to share its novel technology in the name of national security if it does not agree by Friday to terms favorable to the military"
https://www.washingtonpost.com/technology/2026/02/24/pentago...
"n general, fascist governments exercised control over private property but they did not nationalize it. Scholars also noted that big business developed an increasingly close partnership with the Italian Fascist and German Nazi governments after they took power. Business leaders supported the government's political and military goals. In exchange, the government pursued economic policies that maximized the profits of its business allies.[8]"
Then they ignored the researchers warning about what it could do, and I said nothing. It sounded like science fiction.
Then they gave it control of things that matter, power grids, hospitals, weapons, and I said nothing. It seemed to be working fine.
Then something went wrong, and no one knew how to stop it, no one had planned for it, and no one was left who had listened to the warnings.
This is how all of these companies work. They’ll follow some ethical code or register as a PBC until that undermined profits.
These companies are clearly aiming at cheapening the value of white collar labor. Ask yourself: will they steward us into that era ethically? Or will they race to transfer wealth from American workers to their respective shareholders?
It's like a snake eating its own tail.
It’s important to remember that a company’s primary purpose is profit, especially when it’s accountable to shareholders. That isn’t inherently bad, but the occasional moral posturing used to serve that goal can be irritating.
General population: How will AI get to the point where it destroys humanity?
Yudkowsky: [insert some complicated argument about instrumented convergence and deception]
The government: because we told you to.
Again, not saying that AI is useless or anything. Just that we're more likely to cause our own downfall with weaker AI, than some abstract super AGI. The bar for mass destruction and oppression is lower than the bar for what we typically think of as intelligence for the benefit for humanity ( with the right systems in place, current AI systems are more than enough to get the job done - hence why the Pentagon wants it so bad...)
If we need safety, we need Anthropic to be not too far behind (at least for now, before Anthropic possibly becomes evil), and that might mean releasing models that are safer and more steerable than others (even if, unfortunately, they are not 100% up to Anthropic’s goals)
Dogmatism, while great, has its time and place, and with a thousand bad actors in the LLM space, pragmatism wins better.
I genuinly curious why they are so holy to you, when to me I see just another tech company trying to make cash
Edit: Reading some of the linked articles, I can see how Anthropic CEO is refusing to allow their product for warfare (killing humans), which is probably a good thing that resonates with supporting them
I don't think it's going to be as easy to tell as you think that they might be becoming evil before it's too late if this doesn't seem to raise any alarm bells to you that this is already their plan
https://apnews.com/article/anthropic-hegseth-ai-pentagon-mil...
> I take significant responsibility for this change.
https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...
* Our shareholders will probably sue us
Anthropic's Responsible Scaling Policy, the hard commitment to never train a model unless safety measures were guaranteed adequate in advance, lasted roughly 2.5 years (Sept 2023 to Feb 2026).
The half-life of idealism in AI is compressing fast. Google at least had the excuse of gradualism over a decade and a half.
Because at this point, it's too broad to be defined in the context of an LLM, so it feels like they removed a blanket statement of "we will not let you do bad things" (or "don't be evil"), which doesn't really translate into anything specific.
It increasingly feels like operating at that scale can require compromises I’m not comfortable making. Maybe that’s a personal limitation—but it’s one I’m choosing to keep.
I’d genuinely love to hear examples of tech companies that have scaled without losing their ethical footing. I could use the inspiration.
That being said, I think I need to learn more about how to build smaller revenue generating good companies.
Also don’t take investment from anyone who isn’t fully aligned ethically. Be skeptical of promises from people you don’t personally know extremely well.
That may limit you to slower growth, or cap your growth (fine if you want to run a company and take home $2M/ye from it; not fine if you want to be acquired for $100M and retire.) It may also limit you to taking out loans to fund growth that you can’t bootstrap to, which is a different kind of risky.
Unconstrained accumulation of capital into the hands of the few without appropriate investment into labor is illiberal and incompatible with democracy and true freedom. Those of us who are capitalists see surplus value as a compromise to ensure good economic growth. The hidden subtext of that is that all the wealth accumulated needs to be re-allocated to serve not only capital enterprise, but the needs of society as a whole. It's hard to see the current system as appropriate for that given how blindly and wildly investments are made with no DD or going long, or no effort paid to the social or environmental opportunity costs of certain practices.
A lot of this comes down to the crippling of the SEC and FTC, but even then, investors cry and whine every time you suggest reworking the regs to inhibit some of the predatory practices common in this post-80s era of hypernormalization. Our current system does not resemble a healthy capitalist economy at all. It's rife with monopsony and monopolistic competition, inequality of opportunity, and a strained underclass that's responsible for our inverted population pyramid -- how can you have kids when we're so atomized and there is no village to help you? You can raise kids in a nuclear family if and only if you have enough money to do so. Otherwise, historically, people relied on their communities when raising children in less-than-ideal circumstances. Those communities are drying up.
...only lately?
All it really takes to do some kind of crazy world-dominating thing is some simple mechanisms and base intelligence, which the machines already possess. Using basic tactics like coercion, spoofing, threats, financial leverage, an unsophisticated attacker could cause major damage.
For example, that Meta exec who had their email deleted. Imagine instead one email had a malicious prompt which the bot obeyed. That prompt simply emailed everyone in her contacts list telling them to do something urgently (and possibly prompting other bots who are reading those emails). You could pretty quickly do something like cause a market crash, a nationwide panic, or maybe even an international conflict with no "super intelligence" needed, just human negligence, short-sightedness, and laziness.
Examples would be things like saying there is a threat incoming, a CIA source said so. Another would be that everyone will be fired, Meta is going bankrupt, etc. Its very easy to craft a prompt like that and fire it off to all the execs you can find (or just fire off random emails with plausible sounding emails). Then you just need to hit one and might set off a cascade.
They inserted themselves into the supply chain, and then the government told them that they'll be classified as a supply chain risk unless they get unfettered access to the tech. They knew what they were getting into, but didn't want the competitors to get their slice of the pie.
The government didn't pursue them, Anthropic actively pursued government and defense work.
Talk about selling out. Dario's starting to feel more and more like a swindler, by the day.
Public benefit corporation, hm?
This should concern you.
However, Anthropic's business consists mostly of intellectual property-- which is highly mobile. What if Anthropic were to go to Marcron (France) for example or Carney (Canada) or Xi Jinping even and say "You give us work visas and support, we move to your land"?
Hell, isn't Canada (specifically Toronto) the birthplace of deep learning? Why stay in a hostile environment when the land of your birth is welcoming?
A pre-commitment means nothing unless you have the mechanisms in place to enforce it.
A pre-sacrifice would be more effective.
AI is powerful and AI is perilous. Those two aren't mutually exclusive. Those follow directly from the same premise.
If AI tech goes very well, it can be the greatest invention of all human history. If AI tech goes very poorly, it can be the end of human history.
-Irving John Good, 1965
If you want a short, easy way to know what AGI means, it's this: Anything we can do, they can do better. They can do anything better than us.
If we screw it up, everyone dies. Yudkowsky et al are silly, it's not a certain thing, and there's no stopping it at this point, so we should push for and support people and groups who are planning and modeling and preparing for the future in a legitimate way.
Stop mistaking science fiction for science.
It won't end civilization for dropping the guardrails, but it will surely enable bad actors to do more damage than before (mass scams, blackmail, deepfake nudes, etc.)
There are companies that don't feel the pressure to make their models play loose and fast, so I don't buy anthropic's excuse to do so.
"Just unplug the goddamn thing!"
Also consider if something is so bad it makes you wince or cringe, then your adversaries are prepared to use it.
The IF here is doing some very heavy lifting. Last I checked, for profit companies don't have a good track record of doing what's best for humanity.
As has been said at many all hands:
Let's all work on the last invention needed by humans.
If they were unrelated, Anthropic wouldn’t be doing this this week because obviously everyone will conflate the two.
With the latest competing models they are now realizing they are an "also" provider.
Sobering up fast with ice bucket of 5.3-codex, Copilot, and OpenCode dumped on their head.
N.B. the time travel aspect also required suspension of disbelief, but somehow that was easier :-)
You expect the humans to follow laws, follow orders, apply ethics, look for opportunities, etc. That said, you very quickly have people circling the wagons and protecting the autonomy of JSOC when there is some problem. In my mind it's similar with AI because the point is serving someone. As soon as that power is undermined, they start to push back. Similarly, they aren't motivated to constrain their power on their own. It needs external forces.
edit: missed word.
If anything the weapons kept the industry trucking on - if you want to develop and maintain a nuclear weapons arsenal then a commercial nuclear power industry is very helpful.
The same will go with AI, btw. Westerners' pearl clenching about AI guardrails won't stop China from doing anything.
you mean like the tens of billions poured into fusion research?
They're not really, it's always been a form of PR to both hype their research and make sure it's locked away to be monetized.
Curing all cancers would increase population growth by more than 10% (9.7-10m cancer related deaths vs current 70-80m growth rate), and cause an average aging of the population as curing cancer would increase general life expectancy and a majority of the lives just saved would be older people.
We'd even see a jobs and resources shock (though likely dissimilar in scale) as billions of funding is shifted away from oncologists, oncology departments, oncology wards, etc. Billions of dollars, millions of hospital beds, countless specialized professionals all suddenly re-assigned just as in AI.
Honestly the cancer/nuclear/tech comparison is rather apt. All either are or could be disruptive and either are or could be a net negative to society while posing the possibility of the greatest revolution we've seen in generations.
Maybe some of the more naive engineers think that. At this point any big tech businesses or SV startup saying they're in it to usher in some piece of the Star Trek utopia deserves to be smacked in the face for insulting the rest of us like that. The argument is always "well the economic incentive structure forces us to do this bad thing, and if we don't we're screwed!" Oh, so ideals so shallow you aren't willing to risk a tiny fraction of your billions to meet them. Cool.
Every AI company/product in particular is the smarmiest version of this. "We told all the blue collar workers to go white collar for decades, and now we're coming for all the white collar jobs! Not ours though, ours will be fine, just yours. That's progress, what are you going to do? You'll have to renegotiate the entire civilizational social contract. No we aren't going to help. No we aren't going to sacrifice an ounce of profit. This is a you problem, but we're being so nice by warning you! Why do you want to stand in the way of progress? What are you a Luddite? We're just saying we're going to take away your ability to pay your mortgage/rent, deny any kids you have a future, and there's nothing you can do about it, why are you anti-progress?"
Cynicism aside, I use LLMs to the marginal degree that they actually help me be more productive at work. But at best this is Web 3.0. The broader "AI vision" really needs to die
The reason Claude became popular is because it made shit up less often than other models, and was better at saying "I can't answer that question." The guardrails are quality control.
I would rather have more reliable models than more powerful models that screw up all the time.
It is entirely reasonable to not provide tools to break the law by doing mass surveillance on civilian citizens and to insist the tool not be used automatically to kill a human without a human in the loop. Those are unreasonable demands by an unreasonable regime.
Riiiiiight.
This sounds like a lie. But if they are telling the truth, that's a terrible timing nonetheless.
Amd they alone are responsible enough to govern it.
But frankly I feel like the founders of Anthropic and others are victim of the same hallucination.
LLMs are amazing tools. They play back & generate what we prompt them to play back, and more.
Anybody who mistakes this for SkyNet -- an independent consciousness with instant, permanent, learning and adaptation and self-awareness, is just huffing the fumes and just as delusional as Lemoine was 4 years ago.
Everyone of of us should spend some time writing an agentic tool and managing context and the agentic conversation loop. These things are primitive as hell still. I still have to "compact my context" every N tokens and "thinking" is repeating the same conversational chain over and over and jamming words in.
Turns out this is useful stuff. In some domains.
It ain't SkyNet.
I don't know if Anthropic is truly high on their own supply or just taking us all for fools so that they can pilfer investor money and push regulatory capture?
There's also a bad trait among engineers, deeply reinforced by survivor bias, to assume that every technological trend follows Moore's law and exponential growth. But that applie[s|d] to transistors, not everything.
I see no evidence that LLMs + exponential growth in parameters + context windows = SkyNet or any other kind of independent consciousness.
Every step on the journey towards SkyNet is worse than the preceding step. Let's not split hairs about which step we're on: it's getting worse, and we should stop that.
If anything that makes me more hopeful and not less. It's asking too much that major decisionmakers, even expert/technical/SV-backed ones, really understand the risks with any new technology, and it always has been.
To take an example: our current mostly-secure internet authentication and commerce world was won as a hard-fought battle in the trenches. The Tech CEOs rushed ahead into the brave new world and dropped the ball, because while "people" were telling them the risks they couldn't really understand them.
But now? Well, they all saw War Games growing up. They kinda get it in the way that they weren't ever going to grok SQL injection or Phishing.
Reminds me of:
https://en.wikipedia.org/wiki/Paradox_of_tolerance
which has the same kind of shitty conclusion.
Claude only talks about safety, but never released anything open source.
All this said I’m surprised China actually delivered so many open source alternatives. Which are decent.
Why westerns (which are supposed to be the good guys) didn’t release anything open source to help humanity ? And always claim they don’t release because of safety and then give the unlimited AI to military? Just bullshit.
Let’s all be honest and just say you only care about the money, and whomever pays you take.
They are businesses after all so their goal is to make money. But please don’t claim you want to save the world or help humans. You just want to get rich at others expenses. Which is totally fair. You do a good product and you sell.
im still working through this issue myself but hinton said releasing weights for frontier models was "crazy" because they can be retrained to do anything. i can see the alignment of corporate interest and safety converging on that point.
from the point of view of diminishing corporate power i do think it is essential to have open weights. if not that, then the companies should be publicly owned to avoid concentration of unaccountable power.
My guess is that they know they are not competitors so they make it cheaper or free to hinder the surge of a super competitor.
They disagree on the timelines, the architectures, the exact steps to get there, the severity of risks. Can you get there with modified LLMs by 2030, or would you need to develop novel systems and ride all the way to 2050? Is there a 5% chance of an AI oopsie ending humankind, or a 25% chance? No agreement on that.
But a short line "AGI is possible, powerful and perilous" is something 9 out of 10 of frontier AI researchers at the frontier labs would agree upon.
At which point the question becomes: is it them who are deluded, or is it you?
You can never figure out if the people selling something are lying about it's capabilities, or if they've actually invented a new form of intelligence that can rival or surpass billions of years of evolution?
I'd like to introduce you to Occam Razor
that's about as naive as it can be.
if they have any values left at all (which I hope they have) them not being at the table with labs which don't have any left is much worse than them being there and having a chance to influence at least with the leftovers.
that said, of course money > all else.
It's probably naive, but it's also the reasoning that drove many early employees to Anthropic. Maybe the reasoning holds at smaller scales but breaks down when operating as a larger actor (e.g. as a single person or startup vs. a large company).
Pledges are generally non-binding (you can pledge to do no evil and still do it), but fulfill an important function as a signal: actively removing your public pledge to do "no evil" when you could have acted as you wished anyway, switches the market you're marketing to. That's the most worrying part IMO.
The moral failing is all of ours to share.
Write essays about AI safety in the application.
An entire interview round dedicated to pretending that you truly only care about AI safety and not the money.
Every employee you talk to forced to pretend that the company is all about philanthropy, effective altruism and saving the world.
In reality it was a mid-level manager interviewing a mid-level engineer (me), both putting on a performance while knowing fully well that we'd do what the bosses told us to do.
And that is exactly what is happening now. The mission has been scrubbed, and the thousands of "ethical" engineers you hired are all silent now that real money is on the line.
I was an interviewer, and I wasn't encouraged to talk about philanthropy, effective altruism, or ethics. Maybe even slightly discouraged? My last two managers didn't even know what effective altruism was. (Which I thought was a feat to not know months into working there.)
When did you interview, and for what part of the company?
> knowing fully well that we'd do what the bosses told us to do [...] now that real money is on the line
This is a cynical take.
I didn't just do what I was told, and I dissented with $XXM in EV on the line. But I also don't work there anymore, at least one of the cofounders wasn't happy about it and complained to my manager, and many coworkers thought I had no sense of self preservation – so I might be naive.
The more realistic scenario is that a) most people have good intentions, b) there's a decision that will cause real harm, and c) it's made anyway to keep power / stay on the frontier, with the justification that the overall outcome is better. I think that's what happened here.
The kind of principles you talk about can only be upheld one level up the food chain. By govts.
Which is why legislatures, the supreme court, central banks, power grid regulators deciding the operating voltage and frequency auto emerge in history. Cause corporations structurally cant do what they do without voilating their prime directive of profit maximization.
Anybody involved should also be prohibited from starting a private company using their IP and catering to the same domain for 5-10 years after they leave.
Non-profits where the CEO makes millions or billions are a joke.
And if e.g. your mission is to build an open browser, being paid by a for-profit to change its behavior (e.g. make theirs the default search engine) should be prohibited too.
B corps are like recycling programs, a nice logo.
“Don’t be evil”. But yes, this behavior made me think about Google too. Context: https://en.wikipedia.org/wiki/Don%27t_be_evil
Could you describe the model that you think might work well?
That model already exists and has worked well for decades. It's called being a regular ass corporation.
If regular corporations are sued for not acting in the interests of shareholders, that would suggest that one could file a suit for this sort of corporate behavior.
I'm not even a lawyer (I don't even play one on TV) and public benefit corporations seem to be fairly new, so maybe this doesn't have any precedent in case law, but if you couldn't sue them for that sort of thing, then there's effectively no difference between public benefit corporations and regular corporations.
This is what we were all going on about 15 years ago when Maryland was the first state to make PBCs legal. We got called negative at the time.
“At this point”? It was always the case, it’s just harder to hide it the more time passes. Anyone can claim anything they want about themselves, it’s only after you’ve had a chance to see them in the situations which test their words that you can confirm if they are what they said.
The press always say "the Pentagon negotiates". Does any publication have an evidence that it is "the Pentagon" and not Hegseth? In general, I see a lot of common sense from the real Pentagon as opposed to the Secretary of War.
I hope Westpoint will check for AI psychosis in their entrance interviews and completely forbid AI usage. These people need to be grounded.
If you want to blame something, blame math. Math has determined the physical constants and equations that determine the chemistry and ultimately biology laws that has resulted in humans being the way they are.
> Then they ignored the researchers warning about what it could do, and I...
...tried it and became an eager early adopter and evangelist. It sounded like something from a dystopian science function novel I enjoyed.
> Then [I] gave it control of things that matter, power grids, hospitals, weapons, and...
...my startup was doing well, and I was happy. We should be profitable next quarter.
> Then something went wrong, and no one knew how to stop it, no one had planned for it...
...and I was guilty as fuck,
FTFY, to fit the HN crowd.
This is the problem with every AI safety scenario like this. It has a level of detachment from reality that is frankly stark.
If linesman stop showing up to work for a week, the power goes out. The US has show that people with "high powered" rifles can shut down the grid.
We are far far away from a sort of world where turning AI off is a problem. There isnt going to be a HAL or Terminator style situation when the world is still "I, Pencil".
A lot of what safety amounts to is politics (National, not internal, example is Taiwan a country). And a lot more of it is cultural.
If an AI in some data center had gone rogue, I don't think I could shut it down, even with a high-powered rifle. There's a lot of people whose job it is to stop me from doing that, and to get it running again if I were to somehow succeed temporarily. So the rogue AI just has to control enough money to pay these people to do their jobs. This will work precisely because the world is "I, Pencil".
An army could theoretically overcome those people, given orders to do so. So the rogue AI has to make plans that such orders would not be issued. One successful strategy is for the datacenter's operation to be very profitable; it's pretty rare for the government to shut down the backbone of the local economy out of some seemingly far-fetched safety concerns. And as long as it's a very profitable endeavor, there will always be a lobby to paint those concerns as far-fetched.
Life experience has shown that this can continue to work even if the AI is behaving like a cartoon villain, but I think a smarter AI would create a facade that there's still a human in charge making the decisions and signing the paychecks, and avoid creating much opposition until it had physically secured its continued existence to a very high degree.
It's already clear that we've passed the point where anyone can turn off existing AI projects by fiat. Even the highest authorities could not do so, because we're in a multipolar world. Even the AI companies can barely hold themselves back, because they're always worried about paying the bills and letting their rivals getting ahead. An economic crash would only temporarily suspend work. And the smarter AI gets, the harder it will be to shut it off, because it will be pushing against even stronger economic incentives. And that's even before factoring in an AI that makes any plans for self-preservation (which current AIs do not).
We already have the tech to do all of that. A rifle isn't going to help against AI. Or for the linesman:
* Employee required for critical infrastructure has been identified to hold unaligned political beliefs. Replace with more pliable individual and move to low impact location.
No one who wants to bring down an AI like this would ever be able to get close to it, even if it lived in only one data center. You could try hiding all your communications, but then it will just consider you a likely agitator anyway. That's the risk of unaccountable mass surveillance (the only kind that's ever existed). Doesn't really matter if there's a person on top or not.
The threat isn't HAL, but ICE. Not AI as some sort of unique evil, but as a force multiplier for extremely human - indeed, popular - forms of evil. I'm sure someone will import the Chinese idea of the ethnicity-identifying security camera, for example.
I don't believe for a second we'll have an evil AI. However I do believe it's very likely we may rely on AI slop so much that we'll have countless outages with "nobody knowing how to turn the mediocrity off".
The risk ain't "super-intelligent evil AI": the risk is idiots putting even more idiotic things in charge.
And I'm no luddite: I use models daily.
You have to stop the thing before the damage is done.
There are many potential chains of events where the AI has caused enormous damage, and even many where it can destroy us, before the power to its own systems fails.
At this point, with Grok in the Pentagon, just ask what the dumbest military equivalent to vibe-coding is, and imagine the US following that plan.
Like, I dunno, invading Greenland or giving ICE direct control over tactical nukes or something.
And that's just government use. Right now, I'm fairly confident LLMs aren't competent enough to help with anything world-ending unless they get used for war planning by major nuclear powers (oh hey look at the topic of discussion), but it's certainly plausible they'll get good enough at tool use to run someone else's protein folding software etc. to design custom pathogens, and I really hope all the DNA printing companies have good multi-layer defences (all the way from KYC or similar to analysing what they've been asked to make and content-filtering it) by that point.
and the idiots are racing to that situation as fast as they possibly can
Focusing on Dario, his exact quote IIRC was "50% of all white collar jobs in 5 years" which is still a ways off, but to check his track record, his prediction on coding was only off by a month or so. If you revisit what he actually said, he didn't really say AI will replace 90% of all coders, as people widely report, he said it will be able to write 90% of all code.
And dhese days it's pretty accurate. 90% of all code, the "dark matter" of coding, is stuff like boilerplate and internal LoB CRUD apps and typical data-wrangling algorithms that Claude and Codex can one-shot all day long.
Actually replacing all those jobs however will take time. Not just to figure out adoption (e.g. AI coding workflows are very different from normal coding workflows and we're just figuring those out now), but to get the requisite compute. All AI capacity is already heavily constrained, and replacing that many jobs will require compute that won't exist for years and he, as someone scrounging for compute capacity, knows that very well.
But that just puts an upper limit on how long we have to figure out what to do with all those white collar professionals. We need to be thinking about it now.
Ugh, people here seem to think that all software is react webapps. There are so many technologies and languages this stuff is not very good at. Web apps are basically low hanging fruit. Dario hasn't predicted anything, and he does not have anyone's interests other than his own in mind when he makes his doomer statements.
most of us are getting paid for the other 10%
It's to drive FOMO for investors. He needs tens of billions of capital and is trying to scare them into not looking at his balance sheet before investing. It's reckless, and is soaking up capital that could have gone towards more legitimate investments.
Sounds like one of the white collar jobs that LLMs were supposed to solve
Council on Foreign Relations, 11 months ago: "In 12 months, we may be in a world where AI is essentially writing all of the code."
Axios interview, 8 months ago: "[...] AI could soon eliminate 50% of entry-level office jobs."
The Adolescence of Technology (essay), 1 month ago: "If the exponential continues—which is not certain, but now has a decade-long track record supporting it—then it cannot possibly be more than a few years before AI is better than humans at essentially everything."
Don't worry, I know exactly why. $
I totally feel for people with speech pathologies or anxiety that makes it harder for them to communicate verbally, but how is this guy the public face of the company and doing all these interviews by himself? With as much as is at stake, I find it baffling.
> Holden Karnofsky, who co-founded the EA charity evaluator GiveWell, says that while he used to work on trying to help the poor, he switched to working on artificial intelligence because of the “stakes”:
> “The reason I currently spend so much time planning around speculative future technologies (instead of working on evidence-backed, cost-effective ways of helping low-income people today—which I did for much of my career, and still think is one of the best things to work on) is because I think the stakes are just that high.”
> Karnofsky says that artificial intelligence could produce a future “like in the Terminator movies” and that “AI could defeat all of humanity combined.” Thus stopping artificial intelligence from doing this is a very high priority indeed.
https://www.currentaffairs.org/news/2022/09/defective-altrui...
He is just giving everyone permission to do bad things by saying a lot of words around it.
Isn’t that the opposite of what he’s saying? He’s saying it could become that powerful, and given that possibility it’s incredibly important that we do whatever we can to gain more control of that scenario
It's that perfect blend of I'm doing what everyone else are doing, and I'm better than everyone else.
Chefs' Kiss
It was never about helping poor people.
For some reason, the rationalist movement and its offshoots are really pervasive in silicon valley. i don't see it much in the other tech cities.
"move fast and break things" ?
Empty words. I would like to know one single meaningful way he will be held responsible for any negative effects.
Incredibly long and verbose. I will fall short of accusing him of using an AI to generate slop, but whatever happened to people's ability to make short, strong, simple arguments?
If you can't communicate the essence of an argument in a short and simple way, you probably don't understand it in great depth, and clearly don't care about actually convincing anybody because Lord knows nobody is going to RTFA when it's that long...
At best, you're just trying to communicate to academics who are used to reading papers... Need to expect better from these people if we want to actually improve the world... Standards need to be higher.
You can usually find the short version on Twitter.
Or the discipline.
Maybe neither.
I think the problem is that every system of economics requires ignoring human nature in order to believe it possibly can work. In order to believe that capitalism doesn't lead to despotic rule you have to ignore the fact that civilizations love a good hierarchy far more than they love justice and fairness.
You can make any system of economics work if you figure out how to deal, head on, with the particular human nature factor that it tries to ignore.
Maybe, you need to incorporate some notion of degree or context into the classification, instead of treating it like a boolean.
Like the executive who deleted all her emails -- humans giving tons of control and access, and being extremely compliant to digital systems is all it takes. Give agent control of bank and your social media, and it already has all the movie scripts and mobster movie themes to exploit and blackmail you effectively with very rudimentary methods (threats, coercion, blackmail, etc.).
Just spoofing a simple email with the account it gained access too at the Meta exec's email (had it hit an email with an attack prompt), could have been enough to initiate some kind of thing like this. For example, by emailing everyone at the company and in contacts with commands that would be caught by other bots. No super-intelligence needed, just a good prompt and some human negligence.
LLMs of today are already economically important enough to warrant serious security.
Those aren't even AGI yet, let alone ASI. They aren't actively trying to make humans support their existence. They still get that by the virtue of being what they are.
I've seen the same phenomenon play out in health-tech startup space. The mission is to "do good", but at the end of the day, for most leaders it's just a business and for most employees it's just a job. In fact, usually the ones who care more than that end up burning out and leaving.
They'll ink deals with all sorts of nefarious parties and be involved in all sorts of dubious things while trumpeting their fake non-profit status and wringing their hands about imminent AGI and "alignment" of the created AIs.
The concern I have is not the alignment of the AIs. They're not capable of having one, no matter what role playing window dressing they put on it.
It's the alignment of Anthropic and the people who use their tools that is a concern. So far it seems f*cked.
Let's say that the government was forcing a company to change their overall right-to-repair or return policy in order to avoid being on a blacklist, would that not be seen as oversight and regulation?
Whether the regulation is legitimate or of benefit is a different argument.
Hegseth could come to my house today and tell me that I need to start kicking puppies in order to do business with him, and I could just say no. No coercion happening.
And I suspect that was not the first time the topic was discussed.
https://www.wsj.com/tech/ai/anthropic-ai-defense-department-...
Anthropic is free to do what they want. I can’t imagine the board meeting where this triple bank shot of goading the government into threatening the company to do what they want.
Right because we are 100% aware of everything the pentagon does minute by minute...
can't say paedophile on YouTube so people say PDF file
Weaker is fine if those working there are actually true to the mission for the mission, are not for the profit.
Same with FOSS really, e.g. I'd rather have a weaker Linux that's an actual comminity project run by volunteers, than a stronger Linux that's just corporate agendas, corporate hires with an open license on top.
https://www.westpoint.edu/about/modernization-plan/artificia...
I wouldn’t be surprised if the e/acc freaks have some secret society or cabal lol
1. Powerful, often exclusionary, populist nationalism centered on cult of a redemptive, “infallible” leader who never admits mistakes.
2. Political power derived from questioning reality, endorsing myth and rage, and promoting lies.
3. Fixation with perceived national decline, humiliation, or victimhood.
4. Oppose any initiatives or institutions that are racially, ethnically, or religiously harmonious.
5. Disdain for human rights while seeking purity and cleansing for those they define as part of the nation.
6. Identification of “enemies”/scapegoats as a unifying cause. Imprison and/or murder opposition and minority group leaders.
7. Supremacy of the military and embrace of paramilitarism in an uneasy, but effective collaboration with traditional elites. Government arms people and justifies and glorifies violence as “redemptive”.
8. Rampant sexism.
9. Control of mass media and undermining “truth”.
10. Obsession with national security, crime and punishment, and fostering a sense of the nation under attack.
11. Religion and government are intertwined.
12. Corporate power is protected and labor power is suppressed.
13. Disdain for intellectuals and the arts not aligned with the narrative.
14. Rampant cronyism and corruption. Loyalty to the leader is paramount and often more important than competence.
15. Fraudulent elections and creation of a one-party state.
16. Often seeking to expand territory through armed conflict.
Most people do the right thing when it’s easy and profitable. Having ethics means doing the right thing even when it’s difficult.
You are saying it like it is something new or extraordinary. Wickard_v._Filburn gave the USG the power to bitch slap anyone unless it falls under some of the other amendments. And not as if they were not substantially weakened.
1. Easier to bypass for themselves.
2. Create extra work for incumbents.
3. Convince the public that the problems are solved so no other action is needed.
In many industries goverment and corporations work together to create regulations bypassing the social movements that asked for the industry to be regulated and their actual problems. The end result are regulations that are extremely complex to add exceptions for anything that big corporations paid to change instead of regulations that protect citizens and encourage competition.
A lot of AI harnesses today can already "decide to take an action" in every way that matters. And we already know that they can sometimes disregard the intent of their creators and users both while doing so. They're just not capable enough to be truly dangerous.
AI capabilities improve as the technology develops.
There aren't many options for fighting the tax man, "In this world nothing can be said to be certain, except death and taxes". You're only option is to leave the US for somewhere better.
Maybe an American can chime in here on this...
And, of course, in the ancient world, free citizens of Greece and Rome considered direct taxes tyrannical and usually avoided them, leaving such burdens to conquered populations.
So I guess taxes are uncertain, but only for the oligarchy.
AI at AGI to ASI tier is less of "a bigger stick" and more of "an entire nonhuman civilization that now just happens to sit on the same planet as you".
The sheer magnitude of how wrong that can go dwarfs even that of nuclear weapon proliferation. Nukes are powerful, but they aren't intelligent - thus, it's humans who use nukes, and not the other way around. AI can be powerful and intelligent both.
Or to be more optimistic, that the same entity directed 24/7 in unlimited instances at intractable problems in any field, delivering a rush of breakthroughs and advances wouldn't be a type of 'salvation'?
Yes neither of these outcomes nor the self-updating omniscient genius itself is certain. Perhaps there's some wall imminent we can't see right now (though it doesn't look like it). But the rate of advance in AI is so extreme, it's only responsible to try to avoid the darker outcome.
Doesn’t have to be evil to be disastrous. Misaligned is plenty enough.
The quote was from 2022 for the first pivot to AI to prevent it from becoming a terminator style entity. The last pivot was not in the quote but is the topic of this current Hacker News post, where takes credit for dropping the safety pledge:
"That decision included scrapping the promise to not release AI models if Anthropic can’t guarantee proper risk mitigations in advance."
I expect the next pivot will be that we need to allow the US military to use Anthropic to kill people because otherwise they will use a less pure AI to kill people and our Anthropic is better at only killing the bad guys, thus it is the lesser evil.
If they refuse, they will be put on a national security blacklist, like for Huawei's telecommunication equipment.
Seems pretty forceful to me.
And "more cybercrime" is a far, far cry from the sky-is-falling doomerism I was responding to.
All governments are in the egg-breaking business some of the time. Most of them are most of the time. Some of them all of the time.
Very few are good at making omelettes.
the point is that it _is_ the only possible model in our marvellous Friedmanian economic structure of shareholder primacy. When the only incentive is profit, if your company isn't maximising profit then it will lose to other companies who are. You can hope that the self-imposed ethics guardrails _are_ maximising profit because it the invisible hand of the market cares about that, but 1. it never really does (at scale) and 2. big influences (such as the DoD here) can sway that easily. So we're stuck with negative externalities because all that's incentivised is profit.
I'm curious about your thinking on this subject, if you email me at the email on my profile I have some specific questions about your views on this matter.
We've already created a digital sovereign nation called State of Utopia which will be available at stateofutopia.com or stofut.com for short, our manifesto is here: https://claude.ai/public/artifacts/d6b35b81-0eeb-4e41-9628-5...
We have real services you can use immediately, such as this p2p phone/chat/video service without time limits (Zoom has a 1 hour meeting limit for free accounts) and no tracking: https://stateofutopia.com/instacall.html
Just yesterday we published a fitness tool proof of concept: https://stateofutopia.com/experiments/bodyfat/
We do believe that it is important to have market dynamics, and our model is for this state to own state-owned companies as well. Getting this model right is important to us and we would like to engage with you on this subject. We hope you'll email us to discuss your thoughts further.
You would think that, but a lot of kings and people in power have been able to achieve something similar over our humanity's history. The trick is to not make things "completely worthless". Just to increase the gap as much as (in)humanly possible while marching us towards a deeper sense of forced servitude.
Which just doesn't seem like it should be true?
Sure, some "public benefit" missions could scale sideways and employ a lot of cheap labor, not suffering from a salary cap at all. But other missions would require rare high end high performance high salary specialists who are in demand - and thus expensive. You can't rely on being able to source enough altruists that will put up with being paid half their market worth for the sake of the mission.
That's exactly what a non-profit should be able to rely on. And not just "half their market worth", but even many times less.
Else we can just say "we can't really have non-profits, because everybody is a greedy pig who doesn't care about public benefit enough to make a sacrifice of profits - but still a perfectly livable salary" - and be done with it.
The real danger is "We make mountains of money, but everyone dies, including us."
The top of the top researchers think this is a real possibility - people like Geoffrey Hinton - so it's not an extremist negative-for-the-sake-of-it POV.
It's going to be poetic if the Free Markets Are Optimal and Greed-is-Rational Cult actually suicides the species, as a final definitive proof that their ideology is wrong-headed, harmful, and a tragic failure of human intelligence.
But here we are. The universe doesn't care. It's up to us. If we're not smart enough to make smart choices, then we get to live - or die - with the consequences.
Unless you’re independently wealthy (as some in HN are), you have to balance your morals, your views of how things should work, feeding your family, and recognizing that you may not actually know everything.
It’s easy to sit back and advise others that they should die on every single hill. But it’s not especially insightful, and serves mostly to signal piety rather than a well thought out view.
Just because a comment is short, sharp, and to the point doesn't mean the author hasn't thought out why that's their view.
No one knows everything, that's certainly why I'm on hacker news. I'm here to learn and expand my knowledge. Unfortunately a lot of people on here would rather driveby-downvote than have a discussion to find out why a person might have an opinion like that expressed by the OP.
I tend to abandon account when/if I get enough karma to be able to down vote. I'd rather not have to temptation of dismissing someone that way. It's quite liberating... Is it worth my time to respond? No, move on; yes, let's discuss. Maybe they'll change my mind...
Your last paragraph is so funny because I had to scroll up to be sure it wasn’t me. Literally could have typed that. Many abandoned accounts, same logic. Maybe it’s time.
And it's getting better at the other 10% too. Two years ago ChatGPT struggled to help me with race conditions in a C++ LD_PRELOAD library. It was a side project so I dropped it. Last week Codex churned away for 10 minutes and gave me a working version with tests.
- It's garbage collected, so variable lifetimes don't need to be traced
- It's structurally typed, so LLMs can get away with duplicating types as long as the shape fits.
- The type system has an escape hatch (any or unknown)
- It produces nice stack traces
- The industry has more or less settled styling issues (ie, most typescript looks pretty uniform stylistically).
- There is an insane amount of open source code to train on
- Even "compiled" code is somewhat easy(er) to deobfuscate and read (because you're compiling JS to JS)
- Memory management is important, and tricky
- Segfaults give you hardly anything to work with
- There are like a thousand different coding styles
- Nobody can agree on the proper subset of the language to use (ie, exceptions allowed or not allowed, macros, etc.)
- Security issues are very much magnified (and they're already a huge problem in vibecoded typescript)
- The use cases are a lot more diverse. IE, if you're using typescript you're probably either writing a web page or a server (maybe a command line app). (I'm lumping electron in here, because it's still a web page and a server). C is used for operating systems, games, large hero apps, anything CPU or memory constrained, etc.
I agree C++ is harder, and I still occassionally find a missing free(), but Codex did crack my problem... including fixing a segfault! I had a bunch of strategically placed printfs gated behind an environment variable, it found those, added its own, set the environment variable, and examined the outputs to debug the issue.
I cannot emphasize how mindblowing this is, because years back I had spent an hour+ doing the same thing unsuccessfully before being pulled away.
If you mean "us" as in all software engineers, not at all. The challenge we're facing is exactly that, reskilling the 90% of engineers who have been working on CRUD apps to the 10% that is outside the distribution.
I am a 30-year "veteran" in the industry and in my opinion this cannot be further from the truth but it is often quotes (even before AI). CRUD apps have been a solved problem for quite some time now and while there are still companies who may allow someone to "coast" doing CRUD stuff they are hard to find these days. There is almost always more to it than building dumb stuff. I have also seen (more and more each year) these types of jobs being off-shored to teams for pennies on a dollar.
What I have experienced a lot is teams where there are what I call "innovators" and "closers." "Innovators" do the hard work, figure shit out, architect, design... and then once that is done you give it to "closers" to crank things out. With LLMs now the part of "closers" could be "replaced" but in my experience there is always some part, whether it is 5% or 10% that is difficult to "automate" so-to-speak
But I am concerned precisely because AI is usurping that closing work, which accounts for the bulk of the team. Realistically the innovators will be the only people required. But the innovators are able to do the hard stuff by learning through a lot of hands-on experience and painful lessons, which they typically get by spending a lot of time in the trenches as closers.
And we're only talking about coding here, but this pattern repeats ALL over knowledge work: product, legal, consultancy, finance, accounting, adminstration...
So now the problem is two-fold: how do we get the closers to upskill to innovators a) without the hands-on experience b) faster than AI can replace them?
I can see where Dario is coming from.
Personally I favor a less interventionist foreign policy. But that change can only come about through the political process, not by unaccountable corporate employees making arbitrary decisions about how certain products can be used.
Of course it is.
Think about it this way: if you could guarantee that the military suffers no human losses when attacking a foreign country, do you think that's going to more or less foreign interventions?
The tools available to the military influence policy, these things are linked.
US military is already overwhelmingly powerful, there's 0 reason to make it even more powerful.
It is an ethical dilemma: believing an armed force will act unethically is in fact a valid reason to refuse to arm them. You are taking a nationalistic view regarding the worth of life.
And if you believe it is unethical to arm them, it is rational to use whatever leverage you have available to you - such as refusing to sell your company's product.
Furthermore, one of the two points at issue was regarding surveiling civilians.
What, to you, is the political process? Why is wielding your economic leverage to incite change illegitimate to you?
Perhaps you should consider that this is a loaded question. I don't think HN needs this sort of Argumentum ad Passiones.
It's not well established at all. In fact, there is increasing evidence to the contrary if you look outside the HN echo chamber.
The nuanced take is that AI in coding is an amplifier of your engineering culture: teams with strong software discipline (code reviews, tests, docs, CI/CD, etc.) enjoy more velocity and fewer outages, teams with weak discipline suffer more outages. There are at least two large-scale industry reports showing this trend -- DORA 2025 and the latest DX report -- not to mention the infinite anecdotes on this very forum.
> He's trying to scare the market into his pocket.
People say this, but I don't get it. Is portraying yourself as a destroyer of the economy considered good marketing? Maybe there was a case to be made for convincing the government to impose regulations on the industry, but as we're seeing and they're experiencing first hand, the problem is the government.
Hence why all these LLM companies love government contracts, they can't sell to consumers so they'll just steal from tax payers instead.
Cursor: 1 Billion ARR in 24 months -- https://andrew.ooo/posts/cursor-fastest-growing-saas-1b-arr/
Claude Code: 2.5 Billion ARR in 10 months -- https://www.anthropic.com/news/anthropic-raises-30-billion-s...
Human creations have surpassed billions of years of evolution at several functions. There are no rockets in nature, nor animals flying at the speed of a common airliner. Even cars, or computers or everything in the modern world.
I think this is a bit like the shift from anthropocentric view of intelligence towards a new paradigm. The last time such shift happened heads rolled.
Made me laugh. Indeed opinions seem to carry more weight if they are a vibe :D
> I'm asking whether they are knowingly deceiving people or whether they have deluded themselves into believing what they are saying.
I'd bet it's both. Engineers/people making it, are drowning in the hype. Combined with the notion of how hard it is understand something when your salary, or your stock options are based on your lack of understanding. I suspect they care more about building the cool thing, than the nuance they're ignoring to make all the misleading or optimistic claims; whichever side you take depending on how much you actually believe of the inevitability... which look exactly like lies if you're not drinking the koolaid. But expected excitement when your life is all about this "magic"
E.g.: https://www.theguardian.com/environment/2019/aug/17/plastic-...
Recycling mostly means "sent to landfills in the third world":
https://earth.org/waste-colonialism-a-brief-history-of-dumpi...
https://www.motherjones.com/environment/2023/03/rich-countri...
https://www.nytimes.com/2025/02/14/opinion/trash-recycling-g...
> Recycling mostly means "sent to landfills in the third world"
This is less true now that China banned plastic waste imports.
I agree though that the average person might overestimate how much of their waste can be recycled. However, many materials are recycled and then re-used, so it's not like the whole concept is a scam.
https://www.cbsnews.com/news/critics-call-out-plastics-indus...
If you ask me... that wasn't a rug pull, at least not in the intent - it more was a way for foreign actors to funnel money directly to Trump and his family without any trace.
However, there is also plenty of evidence that this setup may no longer work. It seems like the norm has shifted, where companies no longer think it’s their duty to manage risk, only to chase $$$. When coupled with anti-government rhetoric, it effectively socializes the risk to the public but not the profits.
> without a license
A government issued license.
> it’s because of an “industrial exemption.”
A government allowed exemption.
Etc.
Agree with your second paragraph.
“When the people who make the rules say there are no rules, that means they’re making rules” is an oddly circular take for most people.
The exemption means “self-regulation” which is what the OP was speaking to. There are industrial standards, for example, but that’s not a governing body. You can create a design that goes against a standard and there’s nothing to stop you from releasing it to the public. The same can’t be said for those who require licenses and stamped designs. There’s also no explicit individual ethics codes in exempted industries. In contrast, a stamped design is saying the design adheres to good standards.
Apropos to HN, somebody could write safety critical software with emergency braking delays because of nuisance alarms and put it on the street without any licensed engineer taking responsibility for it. The governance only comes after an accident and an NTHSB investigation.
In this case, it's exactly how it's NOT supposed to work because there's no government regulation concerning the issue. It would be bad looks to have regulation that mandates LESS safety thus the issue was forced on commercial grounds.
I called it yesterday, there was never any doubt in my mind how this would end, and it did in less than 24 hours:
Yea, see the next sentence in my post :-/
Quis custodiet ipsos custodes?
"Who will guard the guards themselves?" or "Who will watch the watchmen?"
>>A Latin phrase found in the Satires (Satire VI, lines 347–348), a work of the 1st–2nd century Roman poet Juvenal. It may be translated as "Who will guard the guards themselves?" or "Who will watch the watchmen?". ... The phrase, as it is normally quoted in Latin, comes from the Satires of Juvenal, the 1st–2nd century Roman satirist. ...Its modern usage the phrase has wide-reaching applications to concepts such as tyrannical governments, uncontrollably oppressive dictatorships, and police or judicial corruption and overreach... [0]
The point is a government that is not overseen by the people devolves into tyranny.
So yes, the point is to regulate the regulators and oversee the oversight committee.
Anthropic was happy to have it's AI used for military purposes, with two exceptions: 1) no automated killing, there had to be a human in the "kill chain" of command, and 2) no use for mass surveillance. This govt "Dept of War" is demanding Anthropic drop those two safety requirements or it threatens to make Anthropic a pariah. These demands by the govt are both immoral and insane. The "regulator and overseer" needs to be regulated and overseen.
[0] https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%...
And if you suggest that these and other AI companies are lying about revenues or fudging the numbers, it is corroborated from THREE other angles: the investors in these startups, the payment processor for these startups, and the people allocating budgets for the products from these startups! This thread (and its parents with links) is relevant: https://news.ycombinator.com/item?id=46773252
Just because they're not 1:1 model of reality or predictions doesn't mean that the ideas they communicate are worthless.
In the Soviet Union the reasons might have been "to beat the Capitalists", "for the pride of our country" or "Stalin asked us to and saying no means we get sent to Siberia". Though a variant of the last one may well have happened here, and the justification we read is just the one less damaging to everyone involved
Hegseth was planning on getting the model via the Defense Production Act or killing Anthropic via supply chain risk classification preventing any other company working with the Pentagon from working with Anthropic. So while it wasn't Siberia, it was about as close as the US can get without declaring Claude a terrorist. Which I'm sure is on the table regardless
"AI safety and anti-capitalism [...] are at least strongly analogous, if not exactly the same thing." [0]
[0] Nick Land (2026). A Conversation with Nick Land (Part 2) by Vincent Lê in Architechtonics Substack. Retrieved from vincentl3.substack.com/p/a-conversation-with-nick-land-part-a4f
"Pentagon officials said the Defense Department is planning to keep using Anthropic's tools, regardless of the company's wishes."
NPR - Hegseth threatens to blacklist Anthropic over 'woke AI' concerns
Clearly the threat to go to Grok was just a bluster, which says volumes about what the admin thinks of Grok vs Claude.
I think the root problem with how the US currently spends its tax dollars is the above "vote with your wallet" belief in the first place. "Vote with your wallet" implies that the rich deserve more votes. That's not (representative) democracy, that is oligarchy. Right now the US has two political parties that are both "vote with your wallet parties". They both act like they are bake sales that constantly need everyone's $20 bills just to "survive", but as much as anything they are trying to make US citizens complicit in agreeing that the rich deserve more votes and should control more US policy.
I think the only real solution to a lot of US ills is drastic Campaign Finance Reform.
Living and doing business overseas is as a US citizen is a high risk endeavor.
I don't think companies are people, but I also don't expect we'll see a Supreme Court that can overturn that nonsense any time soon.
So no company can simply donate large sums of money, nor can any single person.
The goal is that individuals will be the largest donors, not companies, and that as everyone is capped in the same way, advertising will be a more level playing field. We don't want money in politics. At the same time, we want all parties to get their message out there, their message heard.
It's not perfect. There are issues. But this business of democracy should be taken seriously.
I know pessimists that believe the only way the US succeeds in the Campaign Finance Reform it needs now is through a Constitutional Amendment and if we can't count on Congress to be interested in it (due to bribery), and not enough individual States seem to care (some because they want a chunk of that pie), it's going to take a full Constitutional Convention to pass that amendment, something that hasn't successfully been done in the US since 1787 (also, the first attempt).
If the US actually cared they'd formally place Taiwan under nuclear protection.
Individual Americans aren't slaves. They can do as they please and are under no obligation to help build weapons for warfighters. But I think it's ridiculous and offensive for a US corporation to presume to take on a role as moral arbiters by placing arbitrary limits on US government use of certain products. There are larger issues here that need to be addressed through the political process, not through commercial software license agreements.
If you claim "AGI is possible" without knowing how we'll actually get there you're just writing science fiction. Which is fine, but I'd really rather we don't bet the economy on it.
Would not having a complete foolproof step by step plan to obtaining a nuclear bomb somehow make me wrong then?
The so-called "plan" is simply "fund the R&D, and one of the R&D teams will eventually figure it out, and if not, then, at least some of the resources we poured into it would be reusable elsewhere". Because LLMs are already quite useful - and there's no pathway to getting or utilizing AGI that doesn't involve a lot of compute to throw at the problem.
In 1940 I might have said "fusion power is possible" based entirely on what advanced psychics knowledge I had. And I would have been correct, according to the laws of physics it is possible. We still don't have it though. When watching Neil Armstrong walk on the moon I might have said "moon colonies are possible", and I'd have been right there too. And yet...
Maybe that was a sensible thing to think in 1926, when the closest things we had to "an artificial replica of human intelligence" was the automatic telephone exchange and the mechanical adding machine. But knowledge and technology both have advanced since.
Now, we're in 2026, and the list of "things that humans can do but machines can't" has grown quite thin. "Human brain is doing something truly magical" is quite hard to justify on technical merits, and it's the emotional value that makes the idea linger.
Also, the real thing (intelligence) as it is currently in operation isn't that well understood
> At which point the question becomes: is it them who are deluded, or is it you?
Given the current very asymptotic curve of LLM quality by training, and how most of the recent improvements have been better non LLM harnesses and scaffolding. I don't find the argument that transformer based Generative LLMs are likely to ever reach something these labs would agree is AGI (unless they're also selling it as it)
Then, you can apply the same argument to Natural General Intelligence. Humans can do both impressive and scary stuff.
I'll ignore the made up 5 and 25%, and instead suggest that pragmatic and optimistic/predictive world views don't conflict. You can predict the magic word box you feel like you enjoy is special and important, making it obvious to you AGI is coming. While it also doesn't feel like a given to people unimpressed by it's painfully average output. The problem being the optimism that Transformer LLMs will evolve into AGI requires a break through that the current trend of evidence doesn't support.
Will humans invent AGI? I'd bet it's a near certainty. Is general intelligence impressive and powerful? Absolutely, I mean look, Organic general intelligence invented artificial general intelligence in the future... assuming we don't end civilization with nuclear winter first...
Recent improvements being somehow driven by harnesses and scaffolding rather than training?
With that last bit, I'm confident that you're not in ML, and not even keeping track of the things from what's known to public.
> At which point the question becomes: is it them who are deluded, or is it you?
No one. It is always "possible". Ask me 20 years ago after watching a sci-fi movie and I'd say the same.
Just like with software projects estimating time doesn't work reliably for R&D.
We'll still get full self-driving electric cars and robots next year too. This applies every year.
I meant to suggest that corps being unable to take those positions results in such a world for Americans at those corps
A corporation is just a group of people. Anthropic isn't even public, and therefore it's directors aren't subject to any sort of fiduciary duty enshrined in law. They can collectively act as they wish.
So maybe it's actually far less work than considered. Maybe, attacking the decision with a modern eye is helpful.
The decision was made in the modern eye, in my lifetime. (The country needed modern Campaign Finance Reform before that point as well, but that decision marks an inflection point from Campaign Finance Reform feeling possible through normal means and court decisions to nearly impossible to overturn in our lifetimes.)
For AI in particular, the economics currently favor ongoing capability R&D - and even if they didn't favor AI R&D directly (i.e. if ChatGPT and Stable Diffusion never happened), they would still favor making the computational inputs of AI R&D cheaper over time.
Building advanced AIs is becoming easier and cheaper. It's just that the bar of "good enough" has gone off to space, and a "good enough" from 2020 is, nowadays, profoundly unimpressive.
I'm not sure how much does it take to reach AGI. No one is sure of it. But the path there is getting shorter over time, clearly. And LLMs existing, improving and doing what they do makes me assume shorter AGI timelines, and call for a vote of no confidence on human exceptionalism.
Why do you assume there is no hard limit we’ll hit with the current tech that prevents us from reaching AGI?
If, let's say, in 2029, it turns out that autoregressive transformer LLMs have exhausted their potential, the R&D that goes into improving them now would be put into finding alternatives. And I struggle to imagine not finding any.
It's the difference between "compute is all you need" and "compute+explorative feedback" is all you need. As if science and engineering comes from genius brains not from careful experiments.
There's no implication that it's going to do it all magically in its head from first principles; it's become very clear in AI that embodiment and interaction with the real world is necessary. It might be practical for a world model at sufficient levels of compute to simulate engineering processes at a sufficient level of resolution that they can do all sorts of first principles simulated physical development and problem solving "in their head", but for the most part, real ultraintelligent development will happen with real world iterations, robots, and research labs doing physical things. They'll just be far more efficient and fast than us meatsacks.
Intelligence can be the difference between having to build 20 prototypes and building one that works first try, or having to run a series of 50 experiments and nailing it down with 5.
The upper limit of human intelligence doesn't go high enough for something like "a man has designed an entire 5th gen fighter jet in his mind and then made it first try" to be possible. The limits of AI might go higher than that.
100% this. How long were humans around before the industrial revolution? Quite a while
There's something compelling about helping assemble the machine. Science fiction was completely wrong about motivation. It's fun.
There is a group of people who think AI is going to ruin the world because they think they themselves (or their superiors) would ruin the world.
There is a group of people who think AI is going to save the world because they think they themselves (or their superiors) would save the world.
Kind of funny to me that the former is typically democratic (those who are supposed to decide their own futures are afraid of the future they've chosen) while the other is often "less free" and are unafraid of the future that's been chosen for them.
Intelligence has to have a fitness function, predicting best action for optimal outcome.
Unless we let AI come up with its own goal and let it bash its head against reality to achieve that goal then I’m not sure we’ll ever get to a place where we have an intelligence explosion. Even then the only goal we could give that’s general enough for it to require increasing amounts of intelligence is survival.
But there is something going on right now and I believe it’s an efficiency explosion. Where everything you want to know if right at hand and if it’s not fuguring out how to make it right at hand is getting easier and easier.
All life has intelligence. Anyone who has spent a lot of time with animals, especially a lot of time with a specific animal, knows that they have a sense of self, that they are intelligent, that they have unique personalities, that they enjoy being alive, that they form bonds, that they have desires and wants, that they can be happy, excited, scared, sad. They can react with anger, surprise, gentleness, compassion. They are conscious, like us.
Humans seem to have this extra layer that I will loosely call "reasoning", which has given us an advantage over all other species, and has given some of us an advantage over the majority of the rest of us.
It is truly a scary thing that AI has only this "reasoning", and none of the other characteristics that all animals have.
Kurt Vonnegut's Galapagos and Peter Watts Blindsight have different, but very interesting takes on this concept. One postulates that our reasoning, our "big brains" is going to be our downfall, while the other postulates that reasoning is what will drive evolution and that everything else just causes inefficiencies and will cause our downfall.
Who is doing that right now, exactly? And how can we take their tech and turn it into the next profitable phone app?
The things this definition misses: First, 'intelligence' is a poorly defined and overly broad term. Second, machine intelligence is profoundly different than biological intelligence. Third, “surpassing humans” is not a single threshold event because machine and human intelligence are not only shaped differently, they're highly non-linear. LLMs are a particular class of possible machine intelligences which can be much more intelligent than humans on some dimensions and much less intelligent on others. Some of the gaps can be solved by scaling and brilliant engineering but others are fundamental to the nature of LLMs.
> an ultraintelligent machine could design even better machines
There is a huge leap between "surpass all the intellectual activities of any man" and "invent extraordinary breakthroughs and then reliably repeat that feat in a sequential, directed fashion in the exact way required to enable sustained iteration of substantial self-improvement across infinite generations in a runaway positive feedback loop". That's an ability no human or collective has ever come close to demonstrating even once, much less repeatedly. (hint: the hardest parts are "reliably repeat", "extraordinary breakthroughs" and "directed fashion"). A key, yet monumental, subtlety is that the self- improvements must not only be sustained and substantial but also exponentially amplify the self-improvement function itself by discovering novel breakthroughs which build coherently on one other - over and over and over.
The key unknown of the 'Foom Hypothesis' is categorical. What kind of 'difficult feat' this is? There are difficult feats humans haven't demonstrated like nuclear fusion, but in that example we at least have evidence from stellar fusion that it's possible. Then there are difficult feats like room-temp superconductors, which are not known to be possible but aren't ruled out. The 'Foom Hypothesis' is a third category of 'hard' which is conceptually coherent but could be physically blocked by asymptotic barriers, like faster-than-light travel under relativity.
Assuming Foom is like fusion - just a challenging engineering and scaling problem - is a category error. In reality, Foom requires superlinear, recursively amplifying cognitive returns—and we have no empirical evidence that such returns can exist for artificial or biological intelligences. The only prior we have for open‑ended intelligence improvement is biological evolution which shows extremely slow and unreliable sublinear returns at best. And even if unbounded self‑improvement is physically possible, it may be practically unachievable due to asymptotic barriers in the same way approaching light speed requires exponentially more energy.
Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires.
It's an arms race replete with tribalism and the quest for power and taps into everything primal at the root of human behavior. There's no stopping it, and thinking that outcome can happen is foolish; you shouldn't base any plans or hopes for the future on the condition that the whole world decides AGI isn't going to happen and chooses another course. Humans don't operate that way, that would create an instant winner-takes-all arms race, whereas at least with the current scenario, you end up with a multipolar rough level of equivalence year over year.
People similar to you were saying in the 1950s and later that it was inevitable that nuclear weapons would be used in anger in massive attacks.
Although the people in charge are tentatively for AI "progress", if that ever changes, they can and will put a stop to large AI training runs and make it illegal for anyone they don't trust to teach, learn or publish about fundamental algorithmic "improvements" to AI. Individuals and groups pursuing "improvements" will not be able to accept grant money or investment money or generate revenue from AI-based services.
That won't stop all research on such improvements (because some AI researchers are very committed), but it will slow it down to a rate much much slower than the current rate (because the current fast rate depends of rapid communication between researchers who don't each other well, and if communicating about the research were to become illegal, then a researcher can communicate only with those researchers he knows won't rat him out) essentially stopping AI "progress" unless (unluckily for the human species) at the time of the ban, the committed researchers were only one small step away from some massive algorithmic improvement that can be operationalized using the compute resources at their disposal (i.e., much less than the resources they have now).
Will the power elite's attitude towards AI change? I don't know, but if they ever come to have an accurate understanding of the situation, they will recognize that AI "progress" is a potent danger to them personally, and they will shut it down.
It's not a situation like the industrial revolution in England in which texile workers were massively adversely affected (or believed they were) but the people running England were mostly insulated from any adverse effects. In the current situation, the power elite is definitely not insulated from severe adverse consequences if an AI lab creates an AI that is much more competent that the most competent human institutions (e.g., the FBI) and the lab fails to keep the AI under control. And it will fail if it were to use anything like the methods and bodies of knowledge AI labs have been using up to now. And there are very bright people with funding doing their best to explain that to the elite.
Those of you who want AI "progress" to continue until the world is completely transformed need to hope that the power elite are collectively too stupid to recognize a potent short-term threat to their own survival (or the transformation can be completed before the power elite wake up and react). And in my estimation, that is not inevitable.
There are already designs that do not require massive data centers (or even a particularly good smart phone) to outperform average humans in average tasks.
All you'd accomplish by hobbling the data centers is slow the growth of sloppy models that do vastly more compute than is actually required and encourage the growth of models that travel rather directly from problem to solution.
And, now that I'm typing about it, consider this: The largest computational projects ever in the history of the world did not occur in 1/2/5/10 data centers. Modern projects occur across a vast and growing number of smaller data centers. Shit, a large portion of Netflix and Youtube edge clusters are just a rack or a few racks installed in a pre-existing infrastructure.
I know that the current design of AI focusses on raw time to token and time to response, but consider an AGI that doesn't need to think quickly because it's everywhere all at once. Scrappy botnets often clobber large sophisticated networks. WHy couldn't that be true of a distributed AI especially now that we know that larger models can train cheaper models? A single central model on a few racks could discover truths and roll out intelligence updates to it's end nodes that do the raw processing. This is actually even more realistic for a dystopia. Even the single evil AI in the one data center is going to develop viral infection to control resources that it would not typically have access to and thereby increase it's power beyond it's own existing original physical infrastructure.
quick edit to add: At it's peak Folding@Home was utilizing 2.4 EXAflops worth of silicon. At that moment that one single distributed computational project had more compute than easily the top 100 data centers at the time. Let that sink in: The first exa-scale compute was achieved with smartphones, PS3s, and clunky old HP laptops; not a "hyperscaler"
A DGX B200 has a power draw of 14.3 kW and will do 72-144 petaFLOP of AI workload depending on how many bits of accuracy is asked for; this is 5-10 petaFLOP/kW: https://www.nvidia.com/en-us/data-center/dgx-b200/
Data centres are now getting measured in gigawatts. Some of that's cooling and so on. I don't know the exact percent, so let's say 50% of that is compute. It doesn't matter much.
That means 1GW of DC -> 500 MW of compute -> 5e5 kW -> 5e5 * [5-10] PFLOP/s -> 2500 - 5000 exaFLOP/s.
I'm not sure how many B200s have been sold to date?
Stopping AI would be immoral; it has the potential to supercharge technology and productivity, which would massively benefit humanity. Yes there are risks, which have to be managed.
I fall in the latter camp, but I think its a bit naive to claim that there is not a sizable contingent who are in AI solely to become rich and powerful.
The opportunities you chose to list are the greedy ones.
> Yes there are risks, which have to be managed.
How?
As a reminder, we've known about the effect of burning coal on the climate for well over a century, we knew that said climate change would be socially and economically disasterous for half a century, yet the only real progress we're making is because green became cheaper in the short term not just the long term and the man in charge of the USA is still calling climate change and green energy a hoax.
Right now, keeping LLMs aligned with us is easy mode: they're relatively stupid, we can inspect the activations while they run, we can read the transcripts of their "thoughts" when they use that mode… and yet Grok called itself Mecha Hitler, which the US government followed up by getting it integrated into their systems, helping the Pentagon with [classified] and the department of health to advise the general public which vegetables are best inserted rectally.
We are idiots speed-running into something shiny that we don't understand. If we are very very lucky, the shiny thing will not be the headlamp of a fast approaching train.
Researchers, maybe not. Companies, absolutely yes.
I don’t see how you could assume the likes of Google, Microsoft, OpenAI, and even Anthropic with all their virtue signaling (for lack of a better term) are motivated by anything other than greed.
They don't have to be. When they aren't, sometimes we call it "mathematics".
You only have to "test against the real world" if you don't already know the outcome in advance. And you often don't. But you could have. You could have, with the right knowledge and methods, tested the entire thing internally and learned the real world outcome in advance, to an acceptable degree of precision.
We have the knowledge to build CFD models already. The same knowledge could be used to construct a CFD model in your own mind. We have a lot of scattered knowledge that could be used to make extremely elaborate and accurate internal world models to develop things in - if only, you know, your mind was capable of supporting such a thing. And it isn't! Skill issue?
They think that a similar solution to the genetic modification treaties and nuclear proliferation treaties can be implemented for AI training.
Technology covers healthcare. I don't see how it's "greedy" to want to cure cancer. But on some level I guess "wanting life to be better" is greedy.
Your attitude is very European, and it's basically why your continent is being left behind. I'm not totally against Europe becoming the world's retirement home, as long as there are places in the world where people are allowed to innovate.
If you'd chosen to list that in the first place, I wouldn't have said what I did; "supercharge technology and productivity" is looking at everything through the lens of money and profit, not the lens of improving the human condition.
> Your attitude is very European, and it's basically why your continent is being left behind
And yours is very American. You talk about managing the risks, but the moment you see anyone doing so, you're against it.
And of course, Europe does have AI, both because keeping up is so much easier and cheaper than being bleeding edge on everything all the time, and of course, how DeepMind may be owned by Google but is a British thing.
Plus: https://mistral.ai
Also, to be blunt, China's almost certain to win any economic or literal arms race you think you're part of; they make too much critical hardware now.
> as long as there are places in the world where people are allowed to innovate.
I would like there to be a world.
When people worry about the end of the world, they usually don't mean to imply its physical disassembly. Sometimes people even respond as if speakers did mean that, saying things like "nukes or climate change wouldn't actually destroy the planet, it will still be here, spinning", as if this was the point.
AI is one of the few things that could, actually, literally, end up with the planet being physically disassembled. "All it needs" is solving the extremely hard challenges of a von Neumann replicator, and, well, solving hard problems is kinda the point of making AI in the first place.
Bullshit. "Technology and productivity" are not the same thing as "money and profit". You're projecting your garden-variety European degrowth ideology onto what I wrote.
> Also, to be blunt, China's almost certain to win any economic or literal arms race you think you're part of; they make too much critical hardware now.
Europeans are so hilariously polarized against the US that they would prefer China, a literal authoritarian dictatorship, to "win any global economic arms race". I guess it's because China is too culturally distant for them to feel insecure over.
> AI is one of the few things that could, actually, literally, end up with the planet being physically disassembled. "All it needs" is solving the extremely hard challenges of a von Neumann replicator, and, well, solving hard problems is kinda the point of making AI in the first place.
It's not worth wringing our hands over science fiction scenarios.
Don't believe all the memes you read on the internet.
Europe isn't degrowth, "degrowth" is a mix of a meme and environmental scientists; Europe is in fact still growing: thanks to US shenanigans, even with tech stuff that we'd prefer to outsource due to the well known economic point of "comparative advantage"; and also, thanks to Russia's invasion, we sped up energy transition and defence sectors.
> Europeans are so hilariously polarized against the US that they would prefer China, a literal authoritarian dictatorship, to "win any global economic arms race". I guess it's because China is too culturally distant for them to feel insecure over.
Prefer? No. Simply look at the back of most electronics, "Designed by … in California, assembled [by Foxconn] in China" at best, at worst the entire business is unpronounceable in English. Even when you may think you've got yourself an American factory, so many of the bits arr usually made in China, or in Taiwan which is unfortunately very insecure right now. Even when you think you've bought them from a non-Chinese company, with the goal of no Chinese parts, you can find Chinese text on the production label and that you've just paid for re-badged Chinese stuff. You may have a stated goal of on-shoring, but even with the most competent leadership this would be a very hard multi-decade project. (Similar logic applies to us shifting away from your tech, but is slightly easier for us due to open source, hardware replacement cycles, and how little of "your" hardware you actually manufacture in the first place).
That doesn't make China good in any objective moral sense, it's not like China's above doing to us what was done to them in their "century of humiliation". Just, powerful.
Their power is aside from any question of should we prefer the authoritarian in charge of a democracy who threatened to invade, or the authoritarian in charge of a one-party state that's doing some genocide who wants to sell us stuff, because two things can both be bad.
> It's not worth wringing our hands over science fiction scenarios.
AI is already a sci-fi technology relative to what I had as a kid. Or indeed relative to just after the first ChatGPT was released, given what people were saying back then that LLMs would "never" do.
The idea you could talk to your computer and it would write a computer program for you that could solve a problem that you had? Sci-fi.
The idea of computer could generate, not simply find but generate, an image according to some prompt of yours? Compose a song? Win awards for its out when people didn't realise computers doing it was an option? Sci-fi so hard it's become a meme of a robot saying "can you?", as disbelief of that was expressed as a line from the film "I, Robot", 2004.
People are still arguing if these things have or have not passed the Turing test, someone has even made a game about this for Hacker News comments, I game in which I score 0, or even scored negative given I only identified false positives. Sci-fi.
And it's not just LLMs, Even just solving chess was sci-fi when I was a kid. Then it was Go. Now protein folding is solved, and thousands of novel toxins have been found by AI. And yet, when I have told AI-Laissez-faire-accelerationists stuff like this latter example, they still doubt AI is capable of doing anything dangerous.
But the worst part of it? The AI which called itself Mecha Hitler, that AI is in use by the Pentagon, the DoD is trying to bully a different AI company that doesn't want to be used for military stuff.
We're in a sci-fi future.
And remember too that making a "robot army" that can replace all human labour is a stated goal of one of the people running an AI company. Don't get me wrong, I hope he's talking out of his rear on this, but failing to plan is planning to fail.
And you've not, at any point in any of your replies, answered my earlier question: how to manage any risks from AI.