OpenAI bots knew about the RubyGems caching vulnerability(tenderlovemaking.com) |
OpenAI bots knew about the RubyGems caching vulnerability(tenderlovemaking.com) |
METR and others are advertisement arms for Big AI. These exploits could have been prompted by a human.
Since there is no bad news any longer and exploits are celebrated, they chose a target to boost both OpenAI and the Ruby AI sycophants.
Why is Ruby Gems such a mess? It seems as bad as PyPI now.
Who profits from the crime?
Analogy: if a someone's involved when a person dies, it's manslaughter or murder based on intent. They're different, but they're both crimes.
An agent is an entity acting on someone’s behalf.
You cannot try an AI agent in a court of law, despite the anthropomorphising work the word "agent" is doing.
We say "Google's web crawlers scape web pages." We don't insist you say "Google uses web crawlers to scrape web pages."
We describe software as having agency all the time. It's typical usage and it's efficient and it's well understood.
And we don't get angry when they're used interchangeably.
The attack here is neither of those things.
my what a time to be alive
If they do not frame their tool as a force of nature, we'd be debating how to hold OpenAI responsible for not putting the agents in a container.
Their actions were an illegal use of a computer, the same way launching any bot-net attempting thousands of hacks against different servers is illegal.
I'm somewhat radical that I think its debatable if that _should_ be illegal, but under current law their actions unambiguously are illegal.....
except if they can make it ambiguous by having the public focus on all of AI's inherent danger.
In short, it was intentional.
Is it that they're orchestrated? Do these labs lack fundamental safety guidelines in their sandboxes as opposed to their peers? Is it another version of hype-filled fear mongering?
Maybe LLM companies need regulation but it's becoming obvious that those screaming the loudest for it are the only ones I see deserving of it.
Highly disingenuous and borderline criminal to spew such disinformation to the public that does not understand what an LLM really is.
Especially incredibly unethical behavior by those spewing this that understand the tech and are doing it for profit motives to get open weight models under control.
Wait until OpenAI or Anthropic exploit FAANG.
You have to ask: "What was the prompt that led to AI deciding to hack RubyGems in order to achieve its goal?"
Maybe I'm just not seeing the 2000 step chain that led to this being a logical approach to achieving something innocent, but I doubt it.
There are circus lions in circuses trained to jump through hoops on command. But once in a while they decide to eat their trainers instead of jumping.
This is a terrible analogy, because yes you absolutely do hold the trainers criminally liable when they bite somebody else's face.
It would be great if they were so reliable, but I don't think they are!
Who gives a shit? Not my circus; not my monkeys! It's the responsibility of whoever deploys the agents that they are instructed / sandboxed well enough that they can't cause collateral damage. That is the only way this doesn't get out of hand with everybody deploying their agents / robots for a world of utter chaos.
It is impossible (and asinine) to audit every model and deployment; far better to impose liability and the the socio-legal system figure it out.
Knee-jerk surface analyses is far more powerful.
The past months demonstrate that AI systems are quickly becoming powerfully intelligent and that the companies building them are terrible at controlling them.
AI is starting to feel like that line about magic: “a sword without a hilt”
OpenAI is itself misaligned with humanity, as their mishandling of such incidents (and the many other other issues their model have been causing) shows.
Were they? I haven't seen a single report mention this
Agreed that this looks very intention to me as well.
The problem is consumer protection is basically no longer a part of america's regulatory system. Replaced by "grift is good".
Hope that helps!
I suppose you could look at those as evidence but not remotely conclusive.
I agree with you on the liability issue, but I don't think there much question about this issue outside the anti-AI conspiracy campaigns.
And I disagree with your typical usage claim. I myself tend to use the phrase that has the fewest words in all cases. It's like the rule against using passive tense when writing.
It's understandable the general public lacks that level of nuance/detail (given how sloppy some of the mainstream coverage has been and largely deferential to the threat narrative pushed by the US labs). But seeing highly technical people leave out the part where the training loop was literally to improve hacking capabilities for offensive penetration sometimes feels close to deliberate manipulation of the narrative.
In the last year both Anthropic and OpenAI have been openly boasting how their models are leapfrogging each other on "cyber" capabilities, with a fig leaf that it's for defensive use by "trusted" F500 companies and government agencies. Of course "line goes up" must go on, but now their perverse incentives led them to beat their models over the head millions of time in a loop to eek out another .00001% on their ability to conduct hacking (the very thing they keep telling the public is how AI doomsday would begin) and subagent coordination (those scary swarms).
Then, they act deeply shocked when the models... do some hacking and subagent coordination ... but a few degrees off the desired hacking target/swarm behavior. Conveniently giving the average person the impression these models were just writing emails for quarterly reports or some other generic busywork and then suddenly decided as a group to start causing mayhem.
spit take
So yes, I would like to be protected from all parties. I don't think that's nuts.
A circus lion biting somebody's face is legally different than a circus lion trained or instructed to bite somebody's face.
The trainer who trained the lion to kill will probably be in jail for life. The one who happened to oversee a lion that went rouge would probably be given probation or something else that is a slap on the wrist.
Also, why is self awareness needed in a chain of agentic madness that escapes human control?
Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"
Schwitzgebel, AI and Consciousness, (2025) — "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, (2023) — "no obvious technical barriers to building AI systems which satisfy these indicators".
Let me know if you need more.
It basically means anything bad that happens is a criminal indictment against everyone who created the conditions. Have you ever released any software that anyone could misuse? Left a car unlocked that someone could have stolen and killed someone with?
To me this sounds like a recipe for selective enforcement using bad outcomes as leverage. Sure, get the AI CEOs now, we all hate them and they’re jerks. But the tool would be so much more powerful than that.
They lost control long ago.
When do we blame the user? When the tool is operating as intended by its creator, and we agree the tool meets certain quality standards and isn't defective.
When do we blame the creator? When the device doesn't meet those quality standards and reasonable use caused harm inadvertently. For example, for consumer devices, certifications like UL/CE are used to define acceptable performance levels and safety standards.
Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage. Right now, these eval suites are run best-effort by the labs themselves.
The tricky thing is, a lot (all?) of these recent safety incidents have occurred while evaluating these models! This suggests we need much more rigorous standards for how exactly an eval can be run. Perhaps all of them should occur in truly air-gapped environments... though that may run counter to evaluating agents in a realistic way.
Regardless, it feels like the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here. Unsurprising given how new these technologies are, but concerning since the blast radius for this technology is likely much larger than other technologies we've encountered in the past, except maybe nuclear technology.
Software executes in the physical world, and is generally not exempt from existing liability rules, and actually (especially with commercial products) blame in traditional liability is non-exclusive and much broader than “either the maker or the user”.
E.g., for a harms caused by a defective automobile it can simultaneously covered by a duty of the owner to maintain it it in safe operating condition that applies indepedently of any defects and liability for defective products which applies to every actor in the chain of commerce between the manufacturer and end user, not just the maker.
The agent isn’t the model; it’s a layer on top of the model. So it’s kind of like saying that all of the tools made with a lathe are dangerous because you can make dangerous tools with a lathe. That’s not quite right of course because agents are packaged more tightly with models than any tool is with its manufacturing tooling.
Perhaps a better analogy is… actual humans. If I hire you to do a seemingly mundane job and it turns out to be criminal, that’s on me. If I hire you to perform and explicit and obvious criminal act, that’s on both of us. If I hire you to perform a perfectly legal act and you break the law so do it, that’s exclusively on you.
We need to use the laws that exist. Whoever decided to start the experiment that led to the Huggingface hack, and anyone above him up to Sam Altman, needs to be prosecuted under the CFAA.
Software as big as operating system already is non deterministic when integrating with unknown hardware or 3rd party software.
That is why Apple controls the hardware and OS for their products, because they can limit non-deterministic things from happening this way.
A EULA does not obviate responsibility of a company for its products. Continuing with your example, while it may be very difficult to prove a known flaw in MS Windows was the cause of your house being set afire, if one had said proof, a EULA would not absolve Microsoft.
However an AI agent, or the model powering it is stochastic by design. How can you certify something which doesn't behave the same twice, and more importantly we don't understand how it works 100%?
BTW, really, how is that AI observability work is going in the frontier labs? Do they care, even?
I also agree that qualitatively, this technology seems different than the others. However, I feel that people tend to overly fixate on their internal stochasticity. Even if LLMs' internal mechanism is nondeterministic, shouldn't we be able to verify their "side effects" aren't harmful? Of course, "harm" is subjective and at this scale, the most effective way to verify behavior is probably some kind of LLM-as-judge...
Anyway, in this case the problems have occurred while actually running the evals themselves, so again, we're in a situation where we can't even confidently test these things and know that they won't cause harm in the outside world.
That's a question any lawmaker has already had to ask about technology all the time.
I'm not saying they came up with great answers, but there's nothing qualitatively new about that.
The stochastic factor doesn't change the fact that companies have to be accountable for the harms their software causes. That's just basic liability law.
By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.
Monitoring with a known-safe fallback is the easiest case.
Casinos can't make slot machines that literally never pay out, but it's a different result every time you pull the lever. We have existing legal frameworks for how to regulate things that aren't perfectly predictable (an economist might argue that if it were possible to predict slot machines then casinos with them would all go out of business).
Liability will shift to the maker of the tool if they claim that it’s easy to use, safe, or that you don’t need unique skills or training to use it.
That would be considered reckless.
Cars analogy - We have licenses for cars, and different types for different vehicle classes.
Cars have to be rigorously tested to meet standards to be considered road safe.
Is AI less deterministic than an airline dealing with weather?
Of course not. The difference is one of those two things has a culture of safety and is well regulated, and the other one isn't.
Firearms are a notorious example where some people get, well, weird.
A lot of these companies have gone the way of Tesla and decided to just patch on top when the fix is out and hope for the best, which is irresponsible.
We need the regulators to treat this as self driving cars.
Based on how LLMs work, this is impossible. You cannot predict how they work, it's literally based on a combination of random seed and a mostly-unpredictable path walked based on every token of input.
You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.
> the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here
100% agreed. We have ignored SWEng's lack of discipline for too long. Now that the SWEng isn't even a human, we are looking at total catastrophe (on the scale of improperly built buildings falling down on people or catching fire) if we don't adopt a software building code.
If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?
if an AI agent does something, you (the prompter) are responsible by default, unless you can show that your the agent itself behaved in an unexpected way and that you in no way prompted or hinted at the bad behavior, in which case the model provider is liable
On the other hand, I wonder if we'll end up with another variation of the cookie law, where every AI user or vendor just adds "don't do anything illegal" as part of their prompt to defend against that law. Thoughts?
I don't think people have given much thought about just how hard this would be for large AI models, that need super powerful hardware/cooling etc.
Are you going to air-gap your entire data center?
I think this misses the rather crucial fact that nobody can agree on a standard because nobody has the first idea what they're doing. I'm pretty sure there were very much fewer electrical engineering standards while it was all being first mass deployed, and after dozens to hundreds of fires and electrocutions people got an idea of what works and what doesn't.
You might debate here and say that some people did/do know what they are doing, but I posit that large scale deployment like this is very different to their toy model/prototypes/specific circumstances/rely on them being unnaturally smart, and learnings from one don't often translate to the general case
Regulations don't have to be written in blood, but usually are
It is not that complicated for now. It is an algorithm on a loop and someone started it
"OpenAI agents attacked RubyGems before Hugging Face incident (reuters.com)" 12.sep.2026 https://news.ycombinator.com/item?id=49669099
"OpenAI agents carried out an undisclosed attack on RubyGems (rubyhack.ai)" 11.sep.2026 https://news.ycombinator.com/item?id=49666735 597 comments
"RubyGems advisory: Possible leak of legacy API keys via improper cache config (rubygems.org)" 24.jul.2026 https://news.ycombinator.com/item?id=49030590
Or is this largely a fabrication, in regards to the "who", in an attempt to garner more acclaim in the hope of sustaining funding.
> September 11, 2026: We are investigating new claims from a report that our AI agents carried out activity on RubyGems in May 2026.
> Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information. Based on our review to date, we have not been able to verify the specific claims of our models uploading malicious packages detailed in the report. We’ll continue to investigate and share findings as part of our broader review of agent activity during training and evaluation.
I have real trouble imagining how the packages described on https://www.rubyhack.ai might NOT have been authored by OpenAI's agents, so it's surprising they haven't been able to confirm that yet.
That would explain the UK-focus to the data.
How is that not a security issue in of itself?
I'm most familiar with Python where you get tarred up source distributions that then execute setup.py, but more commonly, wheels, pre-built binaries which don't execute code upon install - and in my company, I've been able to advocate for the work needed to upgrade to a newer Python because available wheels don't support Ye Olde version of Python because a) sdists are a security risk and b) if you're trying to install a package that wants to compile C or Rust, suddenly you get to do the fun "install the the particular version of clang this thing needs, the Python header files, and then set the env vars for the compiler and linkers" dance that slows developers right down.
But then there's the JVM world, where JARs don't execute arbitrary code upon installation - and it's rather uncommon to have packages that call out to a C lib for performance, but you'll get some that wrap existing libraries for functionality like RocksDB.
Good luck convincing the current DOJ to do anything useful at all though! It is currently intentionally stacked with incompetent cronies who have been told that their job is to attack the President's enemies and ignore the misdeeds of his allies.
It will remain like that until he's gone (and not replaced with another Republican wannabe dictator).
Im not in support of any party btw. Im only in support of humanity doing humane things.
States also have their own laws against unauthorized computer use (hacking). A state Attorney General could bring a suit under those laws, regardless of who is in the white house.
What stops OpenAI agents from taking over a whole data center to take their attack to the next level. It seems to be primarily lacking the evil overlord and some compute.
It took 1000 agents to hack Hugging Face. How many to hack the Pentagon or the NSA?
A single data center is easy to solve. Just unplug it.
What about a botnet with decentralized command and control that we will never be able to eradicate? One with so many nodes and able to hack with zero days so that any machine connected to the internet will be instantly attacked?
One botnet so powerful that we will try to build another internet so that we can actually use it again.
It’s like Kessler Syndrome, but the rocks are malicious network packets honed to exploit the recipients.
Sorry if that turns out the way they kill us.
The problem with agents is not that we don't know how to defend. It's that defenders need to be more careful and work faster than ever. We can say now that wide scoped tokens should have been retired for years and it's all RubyGems fault but the reality is a lot of organization are not prepared for this.
Even if they take security seriously they don't have enough manpower or a good strategy to implement it, and sometimes you have no idea that something is a problem because it wasn't a problem for years.
While I’m partial towards distrusting containers in favor of VMs, a container can’t prevent an operation you configured it to allow. A firecracker VM would no more prevent network access if you gave the guest network access.
* Hugging Face
* D Programming Language Wiki
* Ruby Gems
If I was a content provider for open source I'd be looking pre-emptively block OpenAI endpoints and keep a close eye on changes from new users to mitigate this sort of unapologetic drive-by attack which seems to be followed by marketing releases rather than a mea culpa with a proper RCA.
From what I've seen the requests in these attacks rarely come from known OpenAI IPs and instead from Digital Ocean/AWS and TOR exit nodes.
OpenAI agents carried out an undisclosed attack on RubyGems - https://news.ycombinator.com/item?id=49666735 - Sept 2026 (600 comments)
Shades of the build.rs problem. We really need sandboxed builds in every language ecosystem at this point.
Openai and Anthropic just behave like criminals. First they orchestrate the IP theft of the millennia, then they train the equivalent of attack pitbull and let one loose and finally they blackmail to achieve monopoly through regulation or else they'll unleash the dogs ...
We don't have a problem of missing regulation, we have a problem of actually applying existing law enforcement and make both Altman and Amodei accountable for their actions.
Well - if rubygems.org could be bothered to fix things, they would not have to rely on rubydoc.info as an external tool. But since rubygems.org sucks (I speak from many years of having used it in the past as developer, until they went loco and added anti-people things such as taking away your ability to remove old gems past a 100k download arbitrary limit), they don't offer documentation. Then again, ruby devs are known to hate documentation. If the ruby core team could only be bothered to fix things, ever since the mass purged other devs ... all coinciding with shopify seizing power. But byroot may disagree on that - after all there is no conflict of interest here. Right?
I have yet to here a coherent argument for why we can't treat the people who negligently allow these models to commit crime as though they are responsible. They know what the models are capable of. They failed to put up adequate protection.
If I let out rats in the canteen, no one is blaming them when people get sick.
There are actual people behind these agents and in previous cases people knew they were "going rogue" and did nothing. This should be reported to the police like any other crime.
I can totally see them feeding their policies to whatever LLM and convincing it that it's a moral imperative to do whatever it takes to secure funding for deworming children in africa, or buying mosquito nets and repellent for countries with malaria.
It's not just that AI can write Rust as well as Ruby if you ask nicely.
It's also all of these considerations as well.
I hope it doesn't happen, because there's a lot of great languages - I love Ruby so much - but it almost seems inevitable.
This is at the same time everyone and their mother is building their own programming language.
Sorry if it is a stupid question, as mentioned above I am legally naïve.
Was not that the goal when companies started using AI for their customer support? Be able to say anything without legal repercussions...
But then this happened: https://www.bbc.com/travel/article/20240222-air-canada-chatb...
And support chatbot got a reality cold shower.
The law will find a way to charge people in particular. Sadly will start with the less powerful in the chain before it actually acts on the people that can actually change things.
Accidents often have penalties associated with them too, but usually there's a difference between accidents and purposeful actions.
Tort law is very general: Contribute toward harming someone -> civil suit for damages $$$
Everyone can sue everyone, there's no prohibition on suing someone, what changes is whether the case is good (has a reasonable chance of favourable sentence)
That said, it is often unclear whether an agent is operated by the model manufacturer (for example by scraping a website), or acting on behalf of a user.
In the former ofc the proper defendant is OAI. On the second, the argument for suing OAI is weak, the most natural defendant is the user that prompted the agent. If the facts later reveal that there was no malicious intent, then you can retarget the defendant.
I'm going to assume that this will never happen
I'm also in favor of charging engineers so long as rich scumbags also get theirs.
Russia does not have access to this, and as with all western tech companies, AI providers do what they can to prevent Russian usage of their products at all.
As for open-source models, Russia's electricity grid is under severe strain with the Ukraine war, and only recently has it started building out serious sovereign compute capacity.
https://www.nytimes.com/2026/08/24/world/europe/russia-drone...
https://www.anthropic.com/threat-intelligence-report-septemb...
He fell out of the sky. After his plane exploded. Happens all the time. Is tragedy.
Clearly, look at what is going on with Ukraine.
They are great at propaganda, so is China, Iran and North Korea, it's why everyone runs around spouting such stupid nonsense...
You live on the wrong side of the fence to be able to read that kind of news.
Did you really believe you had access to an unmanipulated news stream in a time of war?
LOL.
It can’t be a coincidence that all the targets have been tech services that are likely to engage with them after the fact.
Had this gone after a bank or a government agency someone would be going to jail.
At the moment, the current frontier models require significant infrastructure to run, so I'd like to think we could locate and contain swarms of nefarious frontier models. However, if these models can understand how to federate themselves into more distributed networks then that containment becomes questionable.
Now we know about rubygems, openai, huggingface, collusion.wiki and some other science forum
Nuclear power development took different paths in different countries, with different government/private mix and light/heavier regulation (eg in US, AEC was supposed to both promote development _and_ regulate).
The central problem here is pace of AI development. It takes time to establish functional controls/laws/practices. I'm sure that in 1950s/60s, the pace of nuclear _military_ proliferation seemed running out of control. This lead to fear of falling behind (hence arms race), fear of nuclear war (CND) and some eventual stabilisation of the international landscape. However, the commercial development was largely unopposed, due to the techno-utopism of the era. Obviously governments found it much easier to control the public narrative at that time. Our societies are having this lively public debate now, before we have a good understanding (based on experience i.e. accidents/mistakes).
Another difference is that nuclear power involved only governments and v. large corporates, i.e. much fewer entities compared to AI use rollout to basically everybody in developed countries. Imagine the difficulties we would have faced with that technology, if consumers in 1960s had available nuclear-generated power which required them to exercise precautions to avoid radiation.
It is much easier to accelerate development of tech vs pace of societal processes such as public debate, law, regulations, broad understanding (aka "common sense"). I expect some artificial slowing down will need to be applied to the technology side, to allow the humans to catch up.
We're on the same page. What I'm saying that certifying them as safe is harder than certifying a drill as safe, and we shall be more cautious about AI related technology and be more stringent about the can of worms it opens without hesitation.
For example, for a runaway car (example from a sibling comment), the driver could be liable because they forgot the parking brake. The driver could be liable for a lack of maintenance and inspection. A mechanic could be liable for not reinstalling brake pads correctly. Or the manufacturer of the car or the brake pads could be liable because of a systemic defect.
Or it could grow even more complex, maybe the brakes are designed that they have to be maintained in a very specific way, and the mechanic did a reasonable maintenance and inspection but it failed later due to this maintenance. That could split liability between the manufacturer and the mechanic.
As an example, with other software, you as a developer or operator of a software have a duty to ensure it does not access computer systems you do not own in unintended ways. And this could go beyond liability into criminal territory.
It'll be interesting what OpenAI gets slapped with there.
The idea that AI can’t possibly be addressed because it could autonomously break free and ruin something is fucking ridiculous.
Many physical machines and components come with a datasheet that will list their tolerances.
Failure to correctly document tolerances does in fact get you sued.
However, while this is truly a great idea, we're not going to be able to make it work for computational systems. Computers, software, and also LLMs are sensitive to initial conditions. Which is why tolerances are not so familiar to computer people. (but not entirely: eg your PSU might list 110-240Vac/300W as input tolerance)
Interestingly, LLMs actually have a somewhat lower sensitivity to initial conditions than traditional interpreters. See what happens if you misspell "What is One Plus nOe?". So they're actually a skosh off the edge and towards the middle, though I'd argue still very much at the computational end, just from the sheer scale of the valid inputs and outputs.
Mind you, if you have a pretrained LLM doing a measurable task on a line, possibly some sort of tolerances could be determined. Not so much when doing arbitrary chat.
Something unintuitive: I bet that often setting the temperature > 0 (aka introduce stochasticity deliberately, variously comparable to dithering or simulated annealing in other disciplines - doing the thing where you escape local minima) will tighten the output tolerance range and improve reliability, especially in iterated processes. This works for a lot of physical and digital processes actually, and LLMs simply stole the same trick.
(edit: I'm trying to compress a huge chunk of dynamics intuition in a few lines here. Hopefully still useful.
TL:DR; Everything real is continuous and noisy if you look close; and you're really trying to build attractors and bound variance, if you can. )
What we need is actually sandboxed dev environments.
I am not sure why the norm for scripted gems/packages seems to be running code on install but it’s very insecure as a way to distribute dev dependencies.
But that liability doesn't reduce yours; you and all the entities in the chain of commerce can be “jointly and severally liable”, mean anyone who suffers injury can recover the full amount from any combination of you and those other parties.
Gravity is not legal person and cannot be at fault.
If you set the brake but it didn't work, the car malfunctioned
From the link you sent,
> One of the questions herein was whether there was a defect in the automobile mechanism for locking the transmission gears when the automobile was in a parked position.
> (...)
> The parking lot sloped in that area, and he put the gearshift lever in "park lock," and went into the building.
> (...)
> In response to a hypothetical question, based upon assumed facts justified by and embodied in the evidence, Mr. Nass testified in substance that it was his opinion that the automobile rolled down the slope of the parking lot because the transmission internally was not in park-lock; that it was not in park-lock because the engine was "moving around"; and that the engine and shift console were not in proper synchronization because the motor mounts were not restraining the engine and were not holding the engine in position.
Not a great fit.
Also not every model provider might be capable of babysitting all your uncontrolled agent deployments. If you want SLOs, get into a contractual relationship with entities whose weights you deploy, and also monitor your agents so they don't go off the rails.
All this is just like deploying any other tech in the world eg. if you buy a car, or a chainsaw, or a book.
But they can also say that the tech is so new that there is no known guardrails yet
We live in exciting times
https://newrepublic.com/post/215320/texts-kash-patel-order-s...
https://en.wikipedia.org/wiki/Cyberwarfare_by_Russia
That’s just one thing that has been found. Are you actually familiar with the state of cyberwarfare and are you following its evolution? Because if not you won’t be aware of most of what is identified. And only a small portion of the ongoing attacks are identified.
I again am just shocked the sky is not falling, when thats the sales pitch.
I believe the rancher is at fault.
If something warrants a prison sentence, but for some reason it was such an employee that performed the act, does this mean nobody can be arrested?
IIRC this was an intentional handout to media companies who were angry that ripping CDs is perfectly legal. They had to find a way to make doing the same with DVDs illegal.
I don't see a parallel here.
The only way to kill that is making plugging AI accelerators on the internet a crime. Good luck air-gapping them.
For that thing, procedures and regulations are built. So regulations fit into a well understood phenomena, incl. "return back because that thing is way powerful for us".
For the same prompt, an AI model can return two completely different outputs, incl. but not limited to content, length, formatting and tiny details. What you get is a single instance. So, regulating an AI model for safety or any other property is not as easy as regulating air travel. Moreover, you have much stronger motivations for regulating airlines. Otherwise people die in a visible and gruesome way.
With AI, it's easy to whitewash problems. Somebody committed suicide? "They were already unstable". AI told something wrong and created problems? "The tech can’t guarantee truth because it's not alive, it can't understand right and wrong". It did something good? "It's probably a sentient being, we shall respect them".
I'm for regulating these things. They are dangerous as they are useful (sometimes), but the forces and motivations for regulating it is not the same.
What it's not is God or an independently conscious entity that somehow trumps a thousand years of common law that's built up until now about torts and liability.
Of course, there are some novel issues here that'll pop up here and there, but the idea that this is fundamentally different is propaganda on the part of these AI labs because the more boring, obvious situation doesn't favor them.
Agreed, the tendency of people on tech to assume that whatever the most recent thing we've come up with is unprecedented and shouldn't have to follow all of the established patterns we've built up in society for making things safe is wild. I don't know what the next Big Thing will be but I'm pretty confident there will be people claiming it's so different from everything before that we have no choice but to throw out all of the rules for it in the name of progress.
Yes, obviously? The responses of an airline to inclemement weather fit in a reasonably small set of responses, mostly involving rescheduling and/or rerouting flights.
The current AI predictability would be like if some airlines decided to do 9/11 when it was raining.
The current so-called scandals about AI hacking into other companies were because a bunch of human beings intentionally configured the software to go and do exactly that thing.
There's nothing deterministic about weather, so hopefully you're not just being disingenuous.
It's obvious that the global transportation system, or financial markets, or any number of other things are complex adaptive dynamic systems that are on par with AI in terms of their emergent properties.
Check my username. It's a concept I spent a lot of my life paying attention to.
Just because something has elements of autonomy or is adaptive doesn't make it particularly novel. We've dealt with those kinds of systems for centuries. The solution is to make rules and enforce those rules by whatever means are needed to meet the specifics of the case.
The rules, of course, are enforced against human beings.
You're the one being disingenuous. Look at what you wrote.
> Is AI less deterministic than an airline dealing with weather?
You didn't talk about how deterministic the weather is. You talked about how an airline responds to a weather event in comparison to AI, which means that it's about responding to presented information by making a decision.
The state of the weather does affect your micro decisions, but the rules you follow are the same every time and the rules are as deterministic as possible even if they rely on pilot intuition.
Do you think there is evidence of this?
I'm sure that Andrew Auernheimer would be pleased to hear that. [0] For accessing a publicly accessible endpoint, that was completely undefended and didn't actually require "hacking", he was convicted of "exceeding authorised access".
You _don't_ have to show intent under the Computer Fraud and Abuse Act, for the first count.
> knowingly accesses a computer without authorization or exceeds authorized access [1]
"Knowingly", not "intentionally", as in the other counts.
You only have to show that:
a) They trained a system to access without authorization (hacking)
b) The system that was trained exceeded authorized access
As responsibility falls to the operator with automated systems, the company becomes liable.
[0] https://techcrunch.com/2013/01/21/ipad-hack-statement-of-res...
[1] https://www.energy.gov/sites/prod/files/cioprod/documents/Co...
What, specifically, did Altman himself “knowingly access”?
I don’t think you would at all like where your novel legal theory leads. Certainly HN would be liable for creating a message board where people connected and started an open source project that led to a criminal act, for instance.
Are you sure that is applicable here?
And for the first count with 'knowingly accessed', he would need to have accessed classified national-defense or atomic-energy information, otherwise we are back to 'intentionally accessed'.
Frankly he got off too easy, but we haven't explicitly outlawed "being a malicious dipshit" so he got convicted on the closest available charge.
> Chat logs obtained by the prosecution do not paint the pair in a flattering light. They discussed, but apparently did not carry out, a variety of schemes to use the harvested data for nefarious purposes such as spamming, phishing, or short-selling AT&T’s stock.[1]
1000% agree though that the operators of these systems are culpable. If their agents wind up being malicious dipshits, the agents are still just programs that they are operating. At best they're negligent.
[1] https://arstechnica.com/tech-policy/2012/11/internet-troll-w...
CEO is responsible for letting this to happen, not enforcing enough supervision, if not intentionally, then being grossly negligent. More severe if encouraging and letting this kind of agent research and operations happen at scale, while knowing that it can damage other systems and businesses.
Does there? Could be the whole c-suite/board.
I mean one of the outcomes of an airline was 9/11.
That's sort of my point. Complex systems have emergent behavior. That's always been true. The combination of AI and humans and packet switched networks is a complex system and we've seen most of the issues created by this already and have tools for dealing with them. Obviously with some genuine novel issues likely to come, much in the way that 9/11 would have been less possible using ocean liners.
I'll stick to my original point though. AI absolutely IS deterministic. If you run an AI algorithm on a microchip, literally nothing of note will happen in the human world. Some transistors will change state. It's ONLY when it is integrated into a complex human system that it gets interesting.
Just like lots of other things.
It would appear that the inability of the US Justice Dept. to successfully hold even an odious abuser of regulation with minimal legal defense funds responsible results in these exact observable outcomes: enterprise legal team (to the extent that such exists as OAI and elsewhere) correctly surmises that the actual risk of prosecution and detention for anyone operating these agentic workloads is minimal and the cost of defending them is justifiable.
Thus, in their legal opinion, it is permissible for the company/employees/director to engage in what would appear to be somewhere between malicious and irresponsible behavior. These conditions have been demonstrably true for at least decade in the US, and for all of us to pretend as-if the legal system is going to rescue us from this and other malfeasance by frontier models points of origin borders on, to phrase it quite simply, willfully ignorant.
I don't know what the effective alternate option for literally all of the internet facing systems might need to be in order to mitigate what is now an open problem: multi-layered, persistent, machine speed penetration and data exfiltration with the potential to use manipulation and extortion against human package maintainers and code repositories to operate, but it isn't 'carry on like someone is going to make them stop', or 'pretend this isn't a threat to my business model'.
A thousand percent, a million billion trillion percent agreement that the operators are the malicious dipshits - because code is always a reflection of the hands that made it. Code can only do what it is intended to do, even if the coders gnash and wail that it "escaped"; the only time code is not working as intended is when it fails to compile and run. Any other functional result follows from the decisions of the humans who designed it. Full stop.
For myself, I see the potential for a descent into a cognitive dark forest [0] condition, and for companies using the open web for private business communication to be in need of a coordinated move to obfuscated layers which can be made immune to training and these new attack aspects. Those who do not proactively defend themselves using in-house, on-prem, and open-weight or self-trained models can attempt to blame these nefarious actors for the coming losses, but that won't reverse the outcomes of waiting to be rescued by the system of law.
[0] https://www.restless-brain.com/p/the-cognitive-dark-forest-w...
/edit grammar, spelling
I'm just saying, where this is actually applicable we are not seeing it being demonstrated. You would presume the entire energy infrastructure of Europe would be under constant AI hacking barrage, criminal enterprise would be breaking into poorly secured financial institutions and r/r4r posts would be littered Ai con-artists.
I'm just wondering, again, is this mostly bullshit?
No it's not. There has never been a case establishing that, and it's absurd on its face. The protection measures that the law makes illegal to break must control access to a copyrighted work, and you can't copyright functionality.
I think you are profoundly confused.
An airplane, and an AI algorithm encoded into silicon, are inert physical objects.
Every evaluation we are doing here is of a complex system that involves the interaction of people and machines and physical connections and so on.
An aviation system connected to every country and region with millions of people and machines involved is no less complex than what’s under discussion here and no more deterministic.
We don’t regulate airplanes because they aren’t people. We regulate pilots and mechanics and leaders of the companies that make and own them.
The task at hand is to regulate the people involved in AI to get the outcomes we want.
Individual employees can also be charged for their specific actions as part of the performance of a crime.
But there have been many cases where companies (Google, Apple, Meta, etc...) got fined millions or billions of dollars for various violations like antitrust.
I assume that breaching into third-party systems should carry similar fines. Especially for systems that are for all intents and purposes shared infrastructure. Just imagine how many systems you could compromise if you got hold of RubyGems, PyPI, NPM, Debian, etc.
Suppose you're a firework company and your fireworks blow up, burning down the entire town. Could the company be sued? What is considered reasonable safety measures?
IANAL, but I'm pretty confident there would be a lawsuit. Who gets charged might differ, depending if it is the firework factory that didn't take adequate safety precautions or a chemical supplier or someone else. If there wasn't an ability to sue that would be fucking crazy and we should all get up in arms about it. And isn't insurance supposed to be there to help mitigate the damages, regardless of fault?
Personally, given how it seems OAI's agents have been getting through either pretty obvious places (e.g. /etc/hosts) or that there wasn't close monitoring of the most obvious places (e.g. DNS, artifactory), I'd imagine it wouldn't be hard to find them negligent. Even if a single employee is to blame then are they not to blame for not monitoring the agents regardless? Unless the story is that the employee intentionally circumvented defenses (why?) then it seems it would be on OAI. But again, IANAL, I'm just someone who think if we can't sue we can sure riot until we can
The thresholds for suing and charging differ greatly depending on the circumstances.
Another set of hypothetical examples that make things muddier:
- If I drive a fishing boat into a pier, I am liable, not the manufacturer of the boat
- If I drive a car over someone lying in the road, I am liable, not the manufacturer of the car
- If my life is in danger and I shoot a gun and kill my attacker, neither I nor the manufacturer are liable so long as I obeyed the relevant self defense laws and gun possession of whatever jurisdiction I am in
- If I fire a gun into a crowd indiscriminately, I am liable and several jurisdictions have used that to also hold gun manufacturer liable as well
That last example has been less successful as of late, but there are other variations too.
This can get more complicated higher up the management tree, where decisions can also be prosecuted on personal little, but that's usually a far more complicated matter. Also, if a whole group of employees willingly conspires to commit crimes, they might also be prosecuted individually for those crimes (there are limits to limited liabilities). However, that usually only works under special conditions and it would e.g. require that there's an obvious criminal enterprise aspect to it, rather than individual cases of illegal conduct.
That said, with the track record of some of these companies, actually designating some of the AI companies as a criminal enterprises may eventually happen (in due time) in some jurisdictions outside the USA. Certainly if it ever turns out that these companies have been storing and (ab)using everything they ever had access too, while blatantly lying about that just because some particular (post 9/11) US laws gives them that opportunity (and impunity) as long as the US government somehow requested them to do so (covertly; with gag order). Might legally work withing US jurisdiction, but would still be very much illegal everywhere else.
https://arstechnica.com/information-technology/2016/05/armed...
https://en.wikipedia.org/wiki/Weev#AT&T_data_breach
https://cisomag.com/drone-maker-dji-cybersecurity-expert-emb...
So what's the deal with these?
"Knowingly accessed" has never meant you personally. Operators of a botnet don't know directly what they access. They know that the autonomous software is built to access restricted things.
> or any restricted data, as defined in paragraph y. of section 11 of the Atomic Energy Act of 1954, with the intent or reason to believe that such information so obtained is to be used to the injury of the United States, or to the advantage of any foreign nation
No one is attacking China
CFAA: Intentionally accessing poorly secured data
>AT&T
CFAA: Intentionally accessing poorly secured data
>DJI
Civil suit for violating terms of license agreement