If you find a good lobbying group with money who can push for it let HN know.

- You shall not embed copyrighted material in your models.

- You shall not bombard every little website in existence with 1 million scraping queries per day.

- You shall not use your political influence to pump and dump your AI (or rocket?) company.

- You shall not imperill the whole IT sector by buying all CPU and memory chips.

These new rules will affect every society directly in a positive way. Thanks.

So, basically, make open-weight models illegal. It's nice for Dario to come out and say this so explicitly.

People get income from one of three places: capital income, labor income, or the welfare state. If this technology truly unlocks a holy panacea of productivity with a commensurate drop in employment then capital’s share of the national income can and should provide for a wider and deeper welfare state. Nothing new need be invented here. Dario’s long and only somewhat organized list of policy interventions makes appropriate preparedness sound like a manic pulling of any and all levers when a simple theory of distribution will suffice.

> AI has become a major commercial technology

>Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety

> AI companies that develop advanced AI models must have strong security standards that protect their model weights

Anyway Dario's financial interests aside. This is an interesting breakpoint for me.

> Second, any response to AI-driven job displacement needs to address both the need to provide for everyone economically, and the need for people to find meaning, purpose, and agency. The latter is ultimately more important

To me this reads as an out of touch statement. I think the majority of people on earth work to keep a roof over their heads. Of course work can be a source of meaning, purpose, and agency, but to call it the more important aspect on a societal level is a sort of rich person like Dario statement to make.

This is a somewhat ironic take from someone who very publicly feuded with the US government about whether their AI could be used for waging war.

Okay, I don't understand how legitimate access is granted then. Surely, Dario isn't saying to ban Sonnet, because I can definitely make it do cyber harm, as most exploits that I've seen in the wild with my own eyes were trivial.

So the only way I see his proposal working is:

- No open weights, AI is centralized in the hands of few

- We get AI-FAA that sets the rules and monitors

- If I want to do a security scan of my codebase, I get a time and scope limited license from AI-FAA that I upload to claude that will allow it to run the security scan in cloud with their models - Claude Mythos Scanner(TM).

Dario's proposal ultimately requires that people lose direct access to inference via API. Is this why they've been building SaaS clones with AI bolted on?

If that arrogance was well placed at least you could somewhat excuse it, but the fact that it is so overtly hypocritical and based on false premises just makes it so much worse.

-C.S. Lewis

They are asking for FAA style preclearance and third party audits. That literally means no new AI startup can emerge. Do they not know that audits cost money?

Protect your own monopoly, protect your customers' regulations. They want strong regulation like the FAA to raise barriers to entry for the foundation models they themselves build, but then why do they want to loosen FDA regulations? While at the same time driving token consumption from their own customers.

They talk about permanent job displacement and UBI. I usually call this "a morally packaged safe landing."

They are doing something unpopular (destroying jobs) and getting criticized for it. But they do not want to be criticized further, and they want to ask for social sympathy. So they claim a 'noble cause' that everyone can sympathize with and that is safe for themselves

AI will generate astronomical productivity gains and capital profits, which AI companies privatize. So why should the social costs be paid by national taxes? In my opinion, something like "We will donate all of our AI companies' revenue for the next 10 years to society" would show genuine sincerity.

Then they say, if we do not develop AI, China will eat our lunch, and they go after China. But is not this really about preventing Chinese dumping, maintaining our own token prices, and asking the world to beat down China so that they can preserve global tech hegemony?

But by blocking China from the CUDA ecosystem, now the CANN ecosystem has emerged, has it not? If China develops techniques that reliably reduce inference costs, who knows how things will turn out then.

Honestly, I like Anthrpic's Claude, but the Anthropic CEO's rhetoric is so stale. It is not that it feels hypocritical. It is that this is just a one dimensional rhetorical tactic that assumes the public is stupid.

I do not think open source is unconditionally good. (It is good, but it can become bad in all situations or all countrie). Open source itself is a barrier for countries outside the Anglosphere when they want to release IT products. Because there is no incentive to buy a product that is worse than an open source alternative. So I do not think everything necessarily has to be open source.

But this (referring to Anthropic's position) seems to treat people like fools. If regulation is needed, shouldn't they also argue that FDA regulation is needed? I wish they would be consistent

I understand why Dario thinks this is crucial, but it's a very dystopian view of the medium-term future.

I'm not an optimist to the point that I believe that AI will lead to global Star Trek-style utopia (although it theoretically could), but ongoing disparity between "allied" and "enemy" powers relating to hardware technology and software models is both not really possible to enforce in the long term, and a pretty dismal state of global affairs even if successful.

I'd be interested in an expert geopolitical opinion on what the long tail of this would really look like in any sort of reasonable reality.

I feel significantly less sympathy for Anthropic's Supply Chain Risk designation if they believe the government should have this power over them. You get what you sign up for.

How much of the policy prescription changes if the exponential is actually just a series of sigmoids[1]?

[1]: https://x.com/ylecun/status/1799064075487572133

This is a massive exaggeration. The advancement in the automation of computer code writing has been impressive and is obviously, at least in the short term, changing the software engineering industry substantially. Most other fields have not been affected to nearly the same degree. Certainly not biology, physics, finance, and law (I don't know enough about the math and translation fields to speak to those).

---

"3. Accelerating AI’s positive impact..."

This whole section is the type of thing that often comes out of the mouths of Silicon Valley tech executives without a pharma background. It indicates a thorough lack of understanding of the realities of pharmaceutical research. What he is describing here is removing many of the solid, evidentiary rules that are in place to make sure that the drugs reaching the market actually work and replacing them with proxy predictions. Look, my least favourite part of the job is the animal testing, and I would be hugely grateful if that could be eliminated from the drug discovery pipeline. People have been trying to do that for a long time. But it's extremely difficult. Biology is very, very complicated. Our understanding of how processes in organisms work are vague and approximative. This is not computer code. Even if Anthropic somehow got all of Big Pharma to hand them their proprietary data, it would only scratch the surface of the understanding that is needed to solve these kind of problems. Due to these realities, the program Amodei is describing here would, effectively, open a floodgate of drugs on the market that don't actually do what they are supposed to and are more likely to have unidentified toxicity.

What makes me doubt that Dario Amodei has really internalized the problem is the lack of humility, the stance that it's just important that the "good guys" keep the technology away from the "bad guys".

If you really want to provide AI with public benefit, you need to prevent power concentration. How? Some unpolished ideas, I'd be happy to hear yours:

- Avoid getting too close to an administration that is openly attacking democracy and is not interested in the benefit of humanity or mutually beneficial cooperation.

- Don't support surveillance. Non-(US-)Americans have human rights and privacy, too. Prepare for a situation where a government tries to convert your compute infrastructure into surveillance infrastructure.

- Support the creation of community data centers. In other words, build data centers together with local communities and make sure they profit from them.

- Advocate for laws that require transparency about resource usage and emissions of data centers.

- If you don't want an AI race, make sure that other countries don't need to fear the US concentrating too much power. Create institutions that can be trusted by other countries, too.

EDIT: I forgot:

- If qualified labor will actually turn out to get devalued, we also need a plan to prevent states from turning into rentier states that don't depend on a well-educated society any longer.

I'm not going to claim that the CEO of pre-IPO company has no incentive to bolster the claims of his tech, but to completely disregard everything he is saying based on that seems awfully binary.

I don't know whether people are just high on copium, spouting "it's just fancy autocomplete" or "only humans can really be creative" on every LLM-related thread, but it is impossible to deny that in a span of a few years we've gone from models that could barely put together a sentence, to something maybe not equivalent to a junior developer, but at least resembling it.

And sure, you can point out every flaw that current day LLMs have, just how everyone pointed out that Stable Diffusion couldn't generate accurate hands (until it could 6 months later!). But the gradient is pretty clear and I am yet to see a well-argued narrative from anyone why scaling laws should fail in the next year or two (by which point it feels like we're going to have a real problem, extrapolating the current trajectory).

I'm very glad this discussion is at least being had, and I wish everyone would get off their high-horse and take things a bit more seriously.

How about 2 years ago? Back then, I wouldn't even trust it to write a 5-line function without making some sort of silly mistake.

Today, I can leave an agent running by itself for 20 or 30 minutes and most of the time, it comes back with a result that's either flawless or can be refined to be good with a few back and forth messages. Maybe I still have to make some high-level decisions ahead of time, but all of the details, including exploring the codebase and figuring out what to do based on that, can be left to the agent. The amount of improvement just in the last 2 years has been staggering.

Now extrapolate how things will look if the trend continues for another 2 or 3 years.

Is this guaranteed to happen? No. But people have been predicting that we're going to hit a wall for a long time now, and we haven't yet. Maybe there's a wall just ahead of us. But maybe there's not -- and the "not" case seems likely enough that we should at least be planning for it.

A highly enthusiastic concussion enthusiast with 10 hands is how one person put it.

These are people in different fields but highly accomplished so I’m feeling comfortable sharing their assessment.

That said Claude Code has a million features like loops that I know exist but never use.

I imagine that spending a lot more time creating an initial spec goes a long way towards independence, I just don't usually do that.

- All of your observations are absolutely dead on

- Yet, we have very very very robust scaling laws that as Dario points out we've had and validated for over a decade. This extends to downstream measures like METR time horizon and compsosite benchmarks like the epoch capability index.

- If you look at where you're at now, which is again dead on, you're looking at a point on a curve that is quite easy to extrapolate, but less easy to tell when exactly on the curve a certain capability or use case undergoes a step change from error rates dropping below a threshold that is hard to anticipate in advance.

So while Dario / other frontier CEOs are understandably unpalatable, they are absolutely spot on with a call out that all of this is bound to happen and happen quickly, and that's without solving several core problems that haven't been solved yet (e.g. continual learning). In 2023, coding agents were just laughable. Yet they followed the same predictable training curves. Anyone looking at the data can see the obvious, and anyone reading newspaper headlines or hacker news comments would get a very different impression.

It's a clever argument because if you question it, you're reminded of the entire history of technological development which is, guess what, exponential.

You're sometimes also dismissed as not understanding the concept of exponentials. This again is clever, as it's baked into the definition that if you don't see it happening, or can't imagine it happening, well that's precisely a tell you're living through an exponential!

All the reasons you might give can be countered with, essentially, "that problem that seems clear today will go away sooner than you can imagine and when it does you'll be on the back foot, so you'd better just assume it will go away and project/plan accordingly".

The trick is entirely that one cannot possibly deny the general power of exponential progress across all of technology, it's almost a law, but it doesn't work in the other direction - no particular local technology is owed exponential growth because of this general pattern. Sometimes things just cap out at merely 'useful' and don't improve much further, no matter how much you want to believe they won't, no matter how steep the progress curve (or, indeed, line) has been up to that point.

To this point the narrative of what these tools can do over these last 3 or 4 years has always been way ahead of the reality. Everyone who works with the tools knows this.

Not everyone wants it to be true, so some will not acknowledge it and will just keep pushing this year-ahead projection as ground truth today. Many (not all) of those people aren't builders, so they don't have to deal with present reality jarring up against this projection of what ought to be possible, they're safe just talking about what should hypothetically be possible and making plans around that that won't be executed for months to years anyway. This keeps the flywheel going, and in fairness, some of the reality has actually caught up in certain ways, so some of those plans will have to some degree worked out which spins the flywheel faster still.

In the end though I just keep thinking: it's been 4 years (as referenced in the post). A lot has happened, the tools are very cool and very useful for certain things. But when I put my head up and look around in the world, even just the software world, nothing's really changed in terms of actual outcomes, in terms of new things appearing or being built that didn't exist 4 years ago. Certainly nothing feels instinctively like it's improved much, subjectively.

Maybe this is what it feels like to be in the knee of a curve of an exponential, but it seems equally reasonable this is just a breakthrough that's kind of improving at a clip you'd expect it to for all the investment put in, but fundamentally is just a new tool that needs to be slowly commercialised in an economically rational way, as we gear up for the next breakthrough which may or may not be related. Who says it must just keep improving forever? This argument never made much sense to me.

Tech people are following a religious belief system whose utopian promise is the all-powerful computer that will end all suffering. I once read an article in reason magazine from over 30 years ago about how an advanced computer in the future will bring everyone who has ever lived back from the deat and let them live in paradise. They were completely serious. Atheists reading this may object to my description of the tech belief system as religious, but I believe it is accuarte. The idea that tech is an imrpovement and will improve people's lives is believed as an act of faith. Tech has its own moral systems based on some form of libertarian progressivism. And in the future, through the inevitable scientific magic of exponential something, a computer will ascend to godhood and judge all mankind for their actions before allowing some into eternal paradise.

To what extent any of this is true is up for debate, but most west coast tech elite are actively working towards this future, and these are the ideas that drive them. It's hard to talk to them about it because this is their woldview, and they imagine everyone to believe what they do.

Training frontier AI models costs money, orders of magnitude more than third-party audits. If you can afford to build the model, you can afford to have it audited.

He is certainly skilled at writing philosophical essays that sound like they make cogent and thoughtful points (and sometimes genuinely do make cogent and thoughtful points), but his company's actions disregard his rhetoric at their best and actively contradict it at their worst. For instance: there was zero pressure on Anthropic to release this model to anyone - they were ostensibly in the lead, which is the exact scenario they said they'd hold back model releases back when they axed their safety policy the instant it came under the slightest amount of economic pressure:

> And it promises to “delay” Anthropic’s AI development if leaders both consider Anthropic to be leader of the AI race and think the risks of catastrophe to be significant. https://time.com/7380854/exclusive-anthropic-drops-flagship-...

Yet this essay proposes this extreme auditing and regulatory administration pipeline that new models are supposed to go through before they release, right after they, themselves, under no pressure, ran a months-long marketing campaign under apocalyptic rhetoric, which they continue to harp on to the point of nerfing/auto-downgrading their model into uselessness for many legitimate tasks that older models had absolutely no issue supporting, while the supposedly extremely dangerous version... can be freely used with no guardrails by their corporate partners.

The hypocrisy here is neither difficult to see nor is it particularly sophisticated, which makes it all the more infuriating.

In the 60 years since, we've barely been able to adapt the 737 to fly longer routes.

I am going to say it. The CEO of a pre-IPO company has extreme incentive to bolster the tech he is selling, to the point where his every action should be viewed as only in service to that goal. Every word he says should be viewed critically through that lense. He is not making this post out of the goodness of his heart, he is doing it in service to the IPO. If it happens to align with your views that's great, but it's still just a marketing stunt to get people with your views to buy in. Don't be fooled. Buy in if you feel it's a good deal, not because of the CEO's marketing.

Personally I feel most of the improvement in the last year comes from tooling/integration (MCPs, realtime documentation access, treesitter support, orchestration) than from the models themselves, in the last year. And still frontier models would routinely come up with bs until you tell them to actually use those tools.

There are people who simultaneously are scared about AI but refuse to believe that AI is the * real deal * and can do everything Dario thinks it can. These are the same people who think its all marketing hype and Dario is "hyping up" before IPO or some lame conspiracy theory.

It is high time people start accepting the real world performance of LLMs and brace themselves instead of hiding behind two contradictory views

1. AI is hype

2. AI is scary

Also, on an unrelated note, why would you have an account for 5 years and only now post your second comment? AI has been an existential threat for years, why only now?

This is a pattern I am seeing all over the place on HN in the last year in AI threads, and I have to admit that I am starting to become paranoid and my feels need some assuaging.