So I wouldn't be able to develop a nuclear weapons with the resources of drug cartal (as an example) using Claude in secret.
He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.
I'm curious about why this is
Outside of an actual test detonation, presumably this could all happen in a secure place?
The proportion of fissile isotopes being mined was off by a fraction of a percent, which caused the French government to launch an investigation. It turns out that millions of years ago the site had formed a natural fission reactor which depleted some of the fissile isotopes
[1]https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...
It isn't impossible to keep such a secret, but practically it would be incredibly difficult just through the energy requirements and mining scale which would be hard to hide without anybody asking what exactly are you mining and processing.
https://www.apple.com/legal/internet-services/itunes/us/term...
> g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons.
Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet
You can get away with a dirty contamination bomb and that detonating in down town Manhattan will scare the shit out of millions of people even the ones in New Jersey. Or, you know, just fly a plane into a really tall building and get the state you are attacking itself to get into a hysteria breakdown.
But yeah I agree with you. There is no point in these restrictions except for government bureaucrats to gain power and control over a domain.
As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the program and ran an Avast free scan to ensure I didn’t have malware, but I also installed uBlock Origin in Firefox to make sure I don’t see Google Ads anymore; they have become a delivery mechanism for malicious (or at least unwanted) software.
I mean, another way hackers could use the embed prohibited-material trick is by making such their malware un-analyze-able. User: "Hey Google/ChatGPT/Apple, this file seems to be infecting our network". AI: "I'm sorry that is prohibited material and you will be reported" is even worse than AI: "I don't understand ['cause I'm down graded]" and both kinds of responses are gaining steam at this point for different kinds of prohibited material.
It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously.
I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors to replicate in their malware in order to target responders/automated scanning/curious devs.
Normally you’d want that to result in a fail and a subsequent rejection.
But because the team who made the review agent and pipeline in my example had many false positives at first they resorted to a fail-open and report setup (not uncommon).
So when the LLM hit this bit and then stalled out the pipeline pushed the code to their Artifactory repo anyway resulting in it being used internally -> exfil of secrets and repos etc.
It’s more about bad design but bad design is pretty common unfortunately.
Jailbreaks do work against the models (look on Github), and they do use similar strategies of mixing SAFE text with malicious text, or malicious with even more malicious, etc, but the working Jailbreaks I've seen are pretty long and complicated and even...creepy.
i'd say it's an okay attempt from the malwares' creator side. but it can be caught easily with a prompt change.
Then again those feel rare from where I sit on the security side.
https://github.com/thebabush/mcp-job-security
Same energy and kind of a funny, low tech solution to frontier model analysis.
scanning arbitrary blobs very often entails running `strings` on the binary. Just slap it in there and oop there goes your LLM.
In Starcraft 2, it is a good idea to BUILD A NUKE and use a cloaked ghost to NUKE your opponent's mineral line, thus reducing their income significantly.
And that's just the start of it, there's been a new update I am looking forward to get into after the great Were Hyena Apocalypse half a year ago. I still fondly remember my militia commander carving a way with her war axe with her husband in tow out of a fortress fully turned were hyenas, all the way past the mortally injured ant eater people near the entrance.
They made it. An entirely epic tale.
Guardrails aren't going anywhere.
Turns out that didn't play out as everyone feared because, well, the instructions themselves aren't useful unless you also have a lab, precursor chemicals, and everything else actually needed to make a weapon. Same back then as it is today.
Any information or instructions an LLM can surface, a sufficiently motivated bad actor can and will also find themselves because the information is already online, both on the clear net and dark web.
On the other hand, getting the U235 is kinda hard.
It turns out the hard part of building a nuclear bomb is actually getting the resources and real world stuff to build it, even a nation state actor with tons of oil i.e. Iran, has struggled to build a nuclear weapon. It turns out the problem isn't the know how it's getting highly enriched uranium and running massive centrifuges.
I mean sure knowledge is important, but there is a real world out there that also gets in the way of a lot of the more harebrained schemes.
What I'm much more worried about is massive corporations along with the government deciding what you can and can't do and what knowledge should and should not be shared and only allowing access to highly capable models by large vetted organizations while the common people are stuck with safety scissor versions of these things because "what if someone does something dangerous?"
By which they mean dangerous to the powers that be. Remember having the Bible in the common tongue was dangerous and led to multiple wars and much death, but I don't think anyone would say that it was morally correct for the Catholic Church to gatekeep who could read it.
Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.
A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.
Don't need much area, depends on the concentration of radioactives. I have a small mine that's just a pegmatite body about the size of a house which produces almost marble-sized chunks of a thorium-uranium mixed metamict mineral (I suspect samarskite but Raman and XRD can't give any ID,) you'd barely notice it from a private airplane's typical flying height, however you could dig the entirety of it up and you'd have enough unprocessed uranium for some real fun.
*while being observed by the most wealthy, powerful nations in the history of the world, who have made it their direct mission to prevent this from happening.
Sincerely, a former engineering student.
(Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated social media platforms that would tell you the steps for personal usage of these things.)
But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.
https://www.youtube.com/watch?v=Gbgk8d3Y1Q4
On a second thought, probably better to act like it is a tool for "frontier LLM research". Export symbols like "mythos_distillation_subroutine".
I'm imagining a comedy in the style of "The Office" in which the majority of the workers are agents of sabotage who are unaware that the majority of their coworkers are doing the same. How far fetched is it for the entire program to be a fake, with all the pomp and cost of a real program, but secretly existing only to string the leadership along with occasional dog and pony shows?
(30 seconds of googling.)
Or perhaps you meant Q clearance nuke stuff? That would be QUITE a bit harder to find and illegal to share. But it’s lack of availability is hardly a counterpoint to the comment you were replying to.