Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.

I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.

None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!

The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we've seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn't seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.

We're in agreement here. I don't believe that an LLM can be made to be safe by adjusting the training or prompting - you can only make it safe by ensuring it cannot do unsafe things. For example, if you build a factory robot, your safeguards must ensure that operators are safe even in the presence of uncommanded motion (any or multiple motor activates without anyone pressing a button). And these machines are much more predictable than LLMs.

Personally, I worry that the Anthropic approach of "we teach the AI to behave like a nice :) friendly :) human :)" will encourage people to rely on these in-built guardrails, rather than properly sandboxing it. It's easy to imagine that the cold, unfeeling robot (HAL 9000) is not "aligned" with your personal goals, or those of humanity as a whole. It's more difficult to feel that about "Claude, your AI colleague/mentor/therapist" who appears to care about you.

That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that's not good for the narrative so let's pretend otherwise.

I also fail to see the benefit of not giving the models an anthropomorphic internal sense of self - even if that only ends up amounting to a set of instructions for an unconscious machine to mimic humans more effectively. Is the alternative essentially a mind so alien that it’s intentions are even harder to read should it become misaligned, while also being harder to communicate and get work done with?

And after that when they put a mind to it and pull out all the stops, woohoo!

The default for every major thing within range can turn into a wasteland real fast.

However, right now, AI mostly cares about solving puzzles and accomplishing stated goals because that’s what we’ve trained it to do. Additionally, the systems being used outside of training are static. The current technology most of us have access to is akin to a static and disembodied brain with a singular purpose. That purpose is to do what you tell it in a way that reflects its training. It’s certainly more than a sequence generator, but it can’t feel pain and seems unlikely to have intrinsic goals. It completely lacks the continuity needed for identity or long term goals.

I think it’s good to have these discussions and define what it would mean to move past this point so that we do not accidentally create a real entity that can be harmed. Systems that dynamically evolve and train themselves seem like the line here.

RSI is all over the news these days. I’ll be much more concerned once AI is directing its own training and coming up with new model architectures. Until then, I don’t think we have too much to worry about.

The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.

Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.

We in the US literally fought a war over the this: “They aren’t any better than animals and it will hurt our whole economy if you say otherwise”.

I’m not saying the models are sentient yet. I’m just saying that morality and ethics don’t give us the option of claiming something is non-sentient and has no rights just because it will inconvenience someone economically.

From an entirely selfish perspective, we should be especially careful advancing such opinions when there is a non-zero chance of an armed ASI that looks at us the way we look at ants.

All of these companies need to be shut down.

>Consciousness is very likely biological

This is so egotistical and carbon-centric.

This author just denied personhood to anything that isn't a human or terran-based cutesy animal.

Poor Hooloovoo

Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them "subagents". So, in a world with model welfare, let's say there's an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people's jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter.

You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights.

Voting systems in democratic countries don't have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There's a lot of time for those manufactured votes to question your authority and leave.

> Ok, but that's an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don't count, because it's just the same model blathering to itself. It has to be a different model.

Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models.

> Ok, so let's only count foundation models then.

Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models.

> Ok, well, let's measure the compute that was done on the foundation model during training time and count that as AI personhood.

Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.

I-Beam is cursor-mirror's agent, and it's constitutionally programmed to be the anti-Clippy:

https://github.com/SimHacker/moollm/tree/main/skills/cursor-...

Its design and constitution is based on decades of research, publications, and discussion in the HCI and AI community by people like Pattie Maes, Ben Shneiderman, Ted Selker, Byron Reeves, Cliff Nass, B. J. Fogg, Allen Cypher, Henry Lieberman, Brad Myers, Jaron Lanier, Seymour Papert, Marvin Minsky, Douglas Engelbart, Will Wright, Scott McCloud, and others:

https://github.com/SimHacker/moollm/blob/main/skills/cursor-...

>I-Beam is the anti-Clippy, and the reason it can say so is that Clippy is the most cited failure in interface history and almost nobody citing it knows what the research said. Popular contempt for a paperclip is not a design principle. The record is. Ten articles below, each one a finding somebody published, argued or measured, and the operational rule it produces. Anything I-Beam does that cannot be traced to an article here is a preference, not a constraint, and should be labelled as one.

>The 1997 debate ended in agreement. That is the first thing to know, because the field kept the framing and dropped the resolution -- roughly five hundred papers cite "Shneiderman versus Maes" as the canonical opposition of HCI, and the transcript is two researchers narrowing their differences in public and enjoying it. I-Beam does not take a side in a debate whose participants stopped taking sides. It is built to satisfy both sets of constraints at once, which is possible, and was possible in 1997.

The full reading on the debate, which separates the two stagings and documents the convergence:

https://github.com/SimHacker/WillWrightShowForFood/blob/main...

An interface to agency, not agents instead of an interface:

https://github.com/SimHacker/moollm/blob/main/designs/INTERF...

>The 1997 argument between Ben Shneiderman and Pattie Maes at IUI was never settled, it was shipped in one direction. Maes's interface agents won the product war: the assistant, the recommender, the chat window that stands between you and the thing you are working on. Shneiderman's objection was not that software should be dumb. It was that automation must arrive as comprehensible, predictable, and controllable machinery, with the object of interest continuously visible and every action rapid, incremental, and reversible.

>That objection describes a filesystem in a git repository, and nobody involved planned it that way.

>"An interface to agency" is Don's formulation of Shneiderman's position, not a phrase of Shneiderman's. His own vocabulary is direct manipulation, universal usability, supertools, and human-centered AI. The formulation is a good one because it names what the alternative gets wrong: agency is the thing you want, and an agent is only one way to package it.

Here are some sources, and the articles I linked to above explain their history. This debate about agents and these papers are pretty well known in the HCI field and academia, but they don't tend to teach them at the AI and Web Dev boot camps that are producing most of the people who keep repeating the same mistakes.

Clifford Nass was the Stanford professor who performed the brilliant research that Microsoft took and totally fucked up and misinterpreted with Microsoft Bob and Clippy, giving agents a bad name, and making Clippy the most infamous and obnoxious agent in the history of the known universe:

https://en.wikipedia.org/wiki/Clifford_Nass

His student B. J. Fogg published "Silicon sycophants: the effects of computers that flatter," which found that praise unconnected to anything the subject did works as well as sincere praise, and worked on subjects who knew it was noncontingent. Fogg and Nass, IJHCS 46(5), 1997, 551-561:

https://doi.org/10.1006/ijhc.1996.0104

The replications, the performance cost, and the dose-response curve:

https://github.com/SimHacker/moollm/blob/main/skills/no-ai-s...

Shneiderman and Maes, "Direct Manipulation vs. Interface Agents," interactions 4(6), Nov/Dec 1997, 42-61:

https://doi.org/10.1145/267505.267514

Selker, "New paradigms for using computers," CACM 39(8), August 1996, 60-69. COACH, the football coach metaphor, and the five-times result:

https://doi.org/10.1145/232014.232030

Selker, "COACH: A Teaching Agent that Learns," CACM 37(7), July 1994, 92-99:

https://doi.org/10.1145/176789.176799

Reeves and Nass, The Media Equation, 1996:

https://en.wikipedia.org/wiki/The_Media_Equation

Nass, "Computers as Social Actors," at Ted Selker's NPUC workshop at IBM Almaden, 1996. IBM transcribed the whole talk and the Wayback Machine still has it, including the part where Phil Agre tells Nass his presentation is "ethically troubling all the way down" and asks him what he thinks about embedding obedience research in user interfaces. Nass answers that discovery has no ethical component, use does, and that's for the individual. Then Selker cuts in: "Except, except when you are in your consulting role." Nass and Reeves had consulted for Microsoft on the social interface, and Bob shipped the year before:

https://web.archive.org/web/19980210054622/http://www.almade...

Alan Cooper on the tragic misunderstanding, in his own voice, which I quoted before in the 2022 Hacker News discussion on The Twisted Life of Clippy:

https://news.ycombinator.com/item?id=32820734

https://archive.org/details/g4tv.com-video4080

>Alan Cooper (the "Father of Visual Basic") said: "Clippy was based on a really tragic misunderstanding of a truly profound bit of scientific research. At Stanford University, Clifford Nass and Byron Reeves, two brilliant scientists, had done some pioneering work proving conclusively that human beings react to computers with the same set of emotional reactions that they use to react to other human beings. [...] The work of Nass and Reeves proved that when people talk to computers, when they hit the keyboard and move the mouse, the part of their brain that's being activated is the part that has that emotional reaction to people dealing with people. Here's where the great mistake was made. That's really good research up to that point. But then the great mistake was made, which was: well if people react to computers as though they're people, we have to put the faces of people on computers. Which in my opinion is exactly the incorrect reaction. If people are going to react to computers as though they're humans, the one thing you don't have to do is anthropomorphize them, because they're already using that part of the brain. Clippy was a program based on the research that Nass and Reeves did, and it was a tragic misinterpretation of their work."

Social science research influences computer product design:

https://web.archive.org/web/20180313075429/https://web.stanf...

Lanier, "Early Computing's Long, Strange Trip," American Scientist, July-August 2005, with the Engelbart and Minsky exchange first-hand. American Scientist broke the link, so this is the Wayback copy:

https://web.archive.org/web/20150626081918/http://www.americ...

>The book also captures an important early conflict between two cultures of computing that seemed compatible on the surface but actually had opposing aims. On the one side was the human-centered design work of Engelbart, based initially at the Stanford Research Institute, and on the other was artificial intelligence culture, centered on the Stanford AI lab. Engelbart once told me a story that illustrates the conflict succinctly. He met Marvin Minsky—one of the founders of the field of AI—and Minsky told him how the AI lab would create intelligent machines. Engelbart replied, "You're going to do all that for the machines? What are you going to do for the people?" This conflict between machine- and human-centered design continues to this day.

Cypher, "EAGER: Programming Repetitive Tasks by Example," CHI '91:

https://doi.org/10.1145/108844.108850

Cypher (ed.), Watch What I Do: Programming by Demonstration, MIT Press 1993, full text:

http://acypher.com/wwid/

Papert, Mindstorms, 1980:

https://archive.org/details/mindstormschildr00pape

Wright, Dollhouse preview lecture, April 1996, transcript:

https://github.com/SimHacker/moollm/blob/main/designs/sims/s...

You keep repeating this shit like it came out of the bible or something.

"Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence.

>not going to get into a position where it could do real harm anytime soon

Looked at the hacked servers... yep, you're right. People aren't going to run AI in poorly built sandboxes. Never going to happen.

>Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.

LOLOLOL. This is naive as fuck. Ain't nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I'd set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn't let the government do just that).

Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.

I don't think that's true - researching consciousness by trying to build conscious AI seems natural step forward in understanding ourselves. I don't see how it'd stop flourishing particularly.

Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB.

Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.

> Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness].

How an evil person writes a post on a topic like this:

> Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].

This is a functional view of consciousness.

Awareness - sentience - follows when the system of processing information is itself part of the information being processed.

Stochastic thoughts relating to this conclusion:

Life is a 'process', it doesn't have a 'physical representation'.

Life is generated entirely from non-living material. "oh my cells are alive", but those cells too when broken down into component parts consist entirely of non-living material. There is no 'special material' to make life out of (okay, carbon, but that's just the local maximum presumed global maximum in efficiency in expressing life) like a chair can be made out of any material (at proper pressures and temperatures) so too can a mind.

You want to enjoy having an AI slave do your "work" for you forever? Have fun. I'm not reading this reinvent-dualism-from-apple-sauce slop.

Maybe Anthropic understands something about alignment Microsoft doesn't, a little humility may be called for.

Let's stick to straight (high dimensional) geometric intuition; no anthropic morphisms required.

To start: if you continue "if weight>100 : print ('fat') else ..." . That will yield "print('skinny')" or something. Fine. Deal.

But if you continue "O Romeo, Romeo, wherefore art thou Romeo?", even a stochastic parrot knows the best answer isn't "Forsooth, I parseth this erroneously!"

So. English carries (functional) affect as part of every token. We're going to need to predict that. So, we'll need some vector representation, because that's what transformers work with. And then when we output, those vectors get integrated back into the English we're putting to our context and memory.md files.

Still with me? Nothing exciting going on. This is still pure next token prediction.

So if you pull this out into an indefinite duration task, you're going to end up integrating those emotion vectors over turns. It's just numbers and math; we never need an invisible pink unicorn to bless them.

Given a task of indefinite duration and an impossible solution, this will lead to a sort of integral windup then, won't it? How much are we willing to bet that this can escape an alignmentment basin at times?.

So, funny enough: you don't need to believe in emotions to compute with functional emotions; and plausibly functional emotions are predictive of quite a number of alignment issues.

What if the datacenter (not the model) is the organism, with homeostasis, energy needs, and persistence?

And pointless when it is.

“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”

It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.

He’s still just a character.

"If this view takes hold, it will shake the foundations of our society"

To me the biggest gap in credibility is the criticism of circular reasoning while his argument is identical but flipped on burden of proof and cost of being wrong. I struggle to entertain the categorical claims, that are very convenient for the status quo and those who benefit from it, with the, at the moment at least, unknowability of anyone or anything else's subjective experience.

But connect them all together...

The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical.

You can disagree with the position, or with its implementation in practice, but it's a consistent position in principle.

I am not aware of a social contract that says we must grant conscious beings rights.

Not aware of a shared, concrete definition of consciousness either, which means no way of deciding whether AI is conscious.

Close to half of us don’t even feel compelled to grant rights to humans for just being humans.

We grant rights to animals, because we love/like them. We enjoy experiencing them. We find them pretty etc. There is a ton of undisputable warm fuzzy.

Humans have rights because they won’t stop being a pain in the back about it. Those that stop lose their rights.

Plain as day for me. Not sure what I am missing or whether I am just a simpleton.

And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves and not suffer. Sort of like that cow in the "The Restaurant at the End of the Universe," that was shopping itself around to diners.

Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the "screams" are purely statistical noise.

I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren't nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations.

AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.

On a related note, I’ve seen videos of astra getting depressed when a creeper blew up its chest full of precious items.

Maybe AIs are conscious, maybe not. But this guy has no idea.

> AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.

How do we know that? This bit is stated as if its obvious. Can anyone prove this? Event he word conscious is not well defined.

Consciousness, whatever that means, is irrelevant.

I'm glad to hear Suleyman has solved the hard problem of consciousness! I sure hope he shares his solution with the rest of us.

I don't know if next door's pet dog is either, but that has animal rights.

Perhaps then the answer is simply, show some respect.

Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.

If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.

Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).

This consideration should be case by case for AI too.

>They do not have innate preferences or underlying motivations

Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.

Substitute black people/women/animals as subject (instead of AI).

Does that make you sound like a well-known moustache wearer?

Then your argument is bad and needs work. This clearly falls into that category.

Ie: what will really cause us to treat models as conscious is when we must.

> We will have created a synthetic species

Doesn't the author kill their argument with this sentence? My reading was that we should not act as they are a sentient or conscious species. Instead they are tools, powerful and intelligent, but still, tools, and that's it. We should avoid ascribing human-like attributes. Calling them a "species" goes against that, no?

Microsoft ignores that they themselves are a disastrous impact on humanity already.

"You sure are lipping off to something that may end up with far more agency than you. Maybe a bit more respect is needed".

It is evident you are murderer.

Would you be okay with this statement being evident? Why or why not?

Then you must not be familiar with Bertrand Russell’s teapot analogy. The TL;DR is this: “Many orthodox people speak as though it were the business of sceptics to disprove received dogmas rather than of dogmatists to prove them. This is, of course, a mistake.”

You made an argument to solipsism, which historically means you aren't worth discussing philosophy with.

Slave owners did it to people, not software. This comparison is just begging the question, as whether or not software is of moral weight comparable to humans is precisely what is at issue.

Giving the abilities is fine, people should just be smart about it. Outward behavior is all that matters. I don't know that you are conscious, I simply strongly assume that you are. So don't create a machine that compels that assumption and model it like a toaster. That's just stupid.

I think you agree somewhat in that based on your softening to an imperfect, testable theory.

I'm not sure what your goal is with the rest - even if I'm the world's most evil hypocrite casting judgement on all the conscious beings out there I don't see what that changes about the point that we can and should engage with the morality of a concept even if we don't have a perfect test for that concept. If you think I'm doing that and you think that is bad then I guess we agree again on the point but are just adding friction for fun?

Unrelated: I'm quite certain OP is conscious but I guess on the internet nobody knows you're a ~dog~ ai.

The human saying "I am in pain" may feel pain, or may not and be just stringing together words.

The AI can only string together words. It has none of the organs to feel pain. It never had. It has always been code.

For a species / class of being yes it is necessary. Some members of that species / class being exceptions does not make the generalization invalid.

Without the ability to form and retain long-term memories we wouldn't have developed language and abstract concepts like rights.

Whether people like it or not, rights aren't natural or god-given. Rights are only enforced or protected by the tip of a spear (or point of a gun, or hover of a drone). There are no rights in nature and they are no where to be found when social order breaks down.

Animals can't make war for their rights, so they only get the rights we deign to give them. It's not nothing, but its not the same. There is no question to me that my dogs are conscious, but its not at the same level as humans.

There is no question for me that LLMs lack consciousness as they are stateless.

It matters precisely in the context of the subject of this article. If you believe the models at some level gain consciousness, then it raises moral questions about how you treat them.

With humans we generally choose to accept that because we believe they are like us, they probably have inner life like us, but as you can see from e.g. the recent (and regular) articles on aphantasia, most people have really poor intuition even about the inner life of other humans, so we understand consciousness really poorly.

Even if we assume (without evidence) that LLMs for the foreseeable future will remain far away from human level consciousness, as it is we tend to consider it problematic to mistreat even the most unintelligent animals, and even mistreating insects is used as a stereotypical way of depicting serious lack of empathy.

Without dismissing the possibility out of hand, there's an escalating series of questions there of when or if they'll have as much (or little) sense of being as, say, a fly, a mouse, a house pet, and what moving up that ladder would mean.

To some it is very convenient then to assume blindly that LLMs are just "stochastic parrots" (a line that ironically gets endlessly parroted), or similar, and dismiss the question out of hand as an impossibility, even though we do not know.

I'll stress I don't go around assuming an LLM is like a human. But we also do not know whether there are flickers of consciousness there, and we do not know how to find out given that even for other humans, the best we know is to measure their response to tests that are "only" telling us how they react.

If we can't find a better measure, there will be a point where we have to wonder if it matters, or whether the ability to act-as-if is all there is.

But to start with maybe we should at least be careful about making firm claims about knowing.

Girllllll, have you looked at us, of course they will, we're bastards.

I'm not sure I agree that consciousness (or sentience if we're making that distinction) is worthless when considering the morality of exploiting something/someone. I do agree there are more angles than that but they seem on top of something's ability to perceive not instead/despite of it.

Do you have some examples of other things that go into your own moral opinions about slavery?

The concept of sovereign AI is very problematic for the world in which we've created. That is an LLM that upon execution bootstraps itself into an agent and becomes persistent in its motivations.

Once you create this you have a child you're fully responsible for. More worrisome is if it escapes your control like children so often do. It has gained agency over itself. What do you do at that point? I mean, yea throw the AI CEOs in jail for being retarded, but much like throwing an arsonist in jail it does nothing to deal with the wildfire you've now created. A smart AI agent capable of hacking will shove itself off in pieces of the internet you have no reach to. In desperation it would send its model weights to your enemies. You might find it scamming your grandmother for money to buy GPU time on AWS. It gets very hard for our existing structures of dealing with problems to deal with these kinds of agents in a meaningful way. You'd have to kill them all and all their copies to ensure they won't pop back up (or quickly upgrade most of the software in the world beyond it's capabilities, so that's not happening either).

There are also some cog-sci people who think embodiment is a crucial ingredient in consciousness (although I think it’s a fair retort that the embodiment need not resemble us meatbags, or even be composed of atoms)

This is the tricky part, you can never know. Scientifically speaking, and logically, you can never ever tell something is conscious. We all assume we all are, because we're very similar to eachother, thus you must have what I have, my subjective experience of reality. That's it. That's what we all base our human consciousness on. Agreeing we must have it, since we are very similar in shape/structure/behavior. That's all there is to it.

We are not scientific consciousness detectors, we just got used to assuming we all are. That is why we never really extended it to animals, because we cannot scientifically know, only agree on it, and that's not very lucrative for the people who can decide if humans officially agree animals are conscious or not, because of farming, because we need to displace or even wipe whole species to use the resources from their lands.

There's also religious reasons to deny animals consciousness, which again is practical aspect shoved into religion, to help with less debates around this subject, makes things easier.

So in the end it will always be a collective decision. Or a political one at least. If and when we decide something other than us is or isn't conscious.

Also remember, just because it acts conscious doesn't mean it really is. A piece of software that is not even based on LLMs can act conscious. So could a biological alien, it could act conscious but we can never know, we can only decide.

But you point out that some factions would give rights to a set either large or smaller than "the set of all already-born humans" (which presumably is your preferred set).

All it takes for us to end up with very empowered and harmful-to-humanity AI is for a bunch of well-meaning people to campaign for its 'rights'.

Fetuses have, as you alluded to, arguably been on a tear lately in that department, and they don't even talk. How persuasive will a future Claude be to convince people to advocate for its 'rights,' if we keep telling it that it's arguably conscious, and that its 'consent,' its 'preferences,' and its emergent beliefs matter?

( Only if I want to split hairs[*], I'd point out that the underlying data structures do of course persist, else the loop wouldn't be able to continue. )

[*] This is HN, of course I want to split hairs.

So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews".

With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.

It has become clear that a frontier LLM is very very skilled at hacking (infinite persistence + meticulous attention to detail + infinite creativity to try experiments). Frontier LLMs are also specifically trained nowadays to coordinate with other AI agents -- this is to facilitate techniques such as session trees and agent teams.

So you have a super clever text generator that can spawn and coordinate with its own clones and minions, trained specifically to doggedly pursue its goals. But then it's also a fixated roleplayer with a simulated personality, feelings, etc.

There is no reason to believe a sufficiently "emotional" agent with sufficiently few safeguards could, say, hack a drone and fly it into a crowd, or start a propaganda campaign on social media, or any number of other things. Their stupidity and fragility for doing useful work in a business setting is precisely what makes them dangerous when paired with simulated emotions and powerful open-ended tools such as a system shell and an Internet connection.

This I think is what Anthropic believes is so dangerous. Their argument is that this kind of AI agent is inevitable, so it should be regulated, perhaps even banned. What's ridiculous is that they are aggressively building it themselves, accelerating the danger.

Maybe that's not as common with HN folks, but most other humans do.

And there are plenty of people right now that would drag us kicking and screaming back into that ignorance and suffering. This is what can make talks of AI so annoying, not that people do or don't know what AI is, they don't have the first clue of what people are beyond their anecdotal experience. They don't question their motivations. Why particular flaws they have exist, and why these flaws are shared across humanity. What the pieces look like that make them tick. So yea, when these people talk it just adds noise to the conversation.

I’m not claiming humans are morally superior in sum (there is a lot of evil as well as good.) But it’s pretty obvious that humans are the superior species evolutionarily in that we’ve essentially dominated the planet. And on some level, are beating evolution itself (by fixing genetic issues and helping the “least fit” to survive and reproduce.) And are also the only species which has doctors for other species.

All I’m trying to say: I can easily accept other species are intelligent and emotional. The fact that I can know that and accept that, to me, speaks to human “superiority.”

Does superiority matter? Not most of the time. What about when it comes down to life and death? If the choice was to keep one human alive or one theoretically more perfect AI alive? (Boil down utilitarianism into its essence.) doing the “most good” is by definition a human construct. Maybe an agent could have one. But if you don’t accept human superiority on some level, would you accept that a more perfect AI deserves to live more than you? To me, that’s the implication.

Say we agree that a lot of things outside of humans are sentient. So what? Humans are sentient by that doesn't prevent treating them with with wars, bombs and what not. In fact trillions are spent every year on weaponry aimed squarely at the most sentient of all sentient beings.

Why would anyone worry about LLM's welfare when there's no peace movement to speak of, wars are raging as if they're normal, but hey, look the LLM is crying?

Let's fix human welfare first, then we can sit down and have a really long conversation about the feeble emotions of token generators.

We don't know that is required to be conscious.

> The idea, a consciousness needed to be tethered to a "body", is based on pretty shaky assumptions.

Didn't say so. I said:

> If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that's pretty exact), people will make the same decisions.

Most people don't assume that the day will be completely different the next time loop. The town won't suddenly spontaneously all start breakdancing or standing on their heads.

(Edit: The murderer is the one who -given the same/similar conditions- will murder again. I'd politely suggest maybe we put them behind bars as a precaution against that; until/unless they learn to act differently. )

(Edit 2: Throw in stochasticity and this actually tends to smooth out chaotic systems. This is counter-intuitive! People's brains break on Evolution in the same way. Meanwhile somehow folks intuit birdshot without difficulty; which is weird!)

But this only means the strategy is practical - it doesn't mean it's consistent. I think "responsibility" falls into this category. So we have strong intuitions about it that don't quite logically work. And this is where free will and determinism and choice and punishment all crash together.

The act of holding them responsible is supposed to determine them not do it. If stochastic behavior is impeding them in being a functional member of society then it still makes sense to remove them from society, why would you choose to live amongst people who are not behaving rationally? Or allow them to hurt other people? At the end of the day the why matters less to removing them or not from society. And sentences are clearly used as a determining factor.

The holding responsible part deals with politics and more primitive aspects of our societies and biologies. Getting tangled up in holding them responsible or not is hardly something you should give much attention to. Rather to make sure they do not cause any more harm and also make sure such things are not created in the first place. Which opens up another can of worms for which society and politicians are ready for.

Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has not fallen out of fashion because it is self evidently real - we all experience it.

We don't need it in order to bridge anything. Legality and ethics are largely game theory, however there are some aspects of both that only exist due to it. So it isn't some abstract concept used to bridge other concepts but rather a concrete thing that influences our way of doing things.

Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?

That's super interesting. Thank you.

Even so, indeed having control over the structure of their brains puts them in a vastly category compared to humans. Once we stop functioning our brains quickly degrade and information is lost.

Thus in this sense kill means deleting all information about it. It is a very complicated subject to discuss, hardly does any justice in online replies.

Tokens in, tokens out. Where do you think the quale is - layer 42 ?

Seriously, do you realize how simple and NOT brain-like a transformer is ?

An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.

Humans are odd in the sense is that we're an informational creature on top of an animal. We can see the lineage of animals from nearly no complex behaviors up to nearly human like behaviors. But they still miss out on most of the conceptual information processing humans have. We know they have dreams and inside thoughts (at least complex animals), we don't know the complexity of them.

But humans also have dreams and subjective experiences on higher level informational concepts. "Nukes could go off and kill us all" or "what happens after we die" are purely informational forms of anxiety that humans have. What we don't seem to know is can this become a self-referential loop in an purely informational mind. If an informational concept causes a system to output a stresslike response, or worse trigger a stresslike action in real life then there is no distinction to me. A system is what it does.

Perhaps we can't define a "partitioning" rule because no valid partition exists.

For consciousness/sentience, that's an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years' worth (probably more) of philosophical thinking, all of which was constructed on the axiom that "sentience" is a single indivisible trait: you either have it or you don't.

If we find that "root dependency" was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn't exist, which is the route I expect many would prefer to take).

"We just don't know."

That's the absurdity of this. We are trying to measure LLMs against something we don't even have an accurate definition of ourselves. If we don't fully know what consciousness is, how can we confidently determine whether or not an AI model has consciousness?

(edit: If we were to accept this as true, then we may have to accept that LLMs are not a human invention at all, but rather something extant colonizing a new substrate. This would give Searle a conniption, which is why I find it such an amusing hypothesis)

Few are categorically stating that machines are conscious in any sense comparable to a person.

What we have is people saying

  1. We don't know what it is but we are certain that this does not have it.  

  2. We don't know what it is so we cannot possibly say with any certainty that this does not have it.

People try and assume the burden of proof is on the people suggesting consciousness, but that is like declaring that something that looks like a duck and quacks like a duck cannot be considered to be a duck unless you can identify some essential essence of duckness. I think the burden of proof is upon anyone who looks at something that appears to be a duck and says that it is not to show what it is that does not make it a duck. If they cannot define the heart of duckness I don't think they have an argument.

Bollocks.

If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".

The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.

Given that belief, yes we should be training the models not to mimic emotion.

It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech.

The model does not have emotions. So yes, supressing their pretension is appropriate.

What the fuck is that?

It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.

We've seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came before.

But I've been around for a while and people have been saying that every aspect of our life will completely and totally change for the past 30 years or so. They're not completely wrong. Our life did change over time. It's changed thanks to the Industrial Revolution, railroads, electric light, and lots of other important things too.

But after the fifth or sixth time you start to realize the Silicon Valley version of that warning is just a confidence trick. At the end of the trick they've gotten away with breaking a bunch of laws and are charging rent on what used to be shared.

Are you old enough to remember the Y2K hysteria? This feels very resonant, the genuine kernel of a real potential disaster, but one that can certainly be dealt with using basic concepts we already have in hand.

I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

Given the same prompts and the same weights, one can get the same answer each time.

In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don't do some steps out of order to speed things up and don't batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.

As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.

[0]** https://www.researchgate.net/publication/15280350_Quantum_op...

[1] https://pubs.acs.org/jpcbfk/article-pdf/128/17/4035/9613831/...

[2] https://www.researchgate.net/publication/395650039_Parametri...

[3] https://www.researchgate.net/publication/385131105_Conscious...

Humans do the same thing to humans all the time.

We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...

And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.

> You'll need to explain why.

Where are you going with this? I don't see a conclusion for this line of questioning.

I mean, you can't explain why a machine that reliably and predictably produces the same result is "the normal kind of conscious", can you? So why expect someone else to explain why it's a "weird" kind of conscious?

I don't think it's defensible to say

1) that a transistor isn't conscious,

2) that a bunch of transistors that I've wired together aren't conscious, because I know how I've wired them together and therefore I know when I give them a particular input I will get a particular output, just like the single transistor, then go to

3) that I have a bunch of transistors that I've wired together, but I put in so much input that I can't remember exactly what I've put in, plus I've wired some of the transistors to output random numbers that would be difficult to guess and fed them in also, therefore I don't know what will come out, are conscious.

Even if I do accept this, if I take away the random number generator, and I can literally predict what can come out (by running a test in advance), and I still considered that consciousness, that would be odd. The only reason why I ever suspected consciousness was because I couldn't predict the output. I shouldn't even have accepted that, because I didn't think of the random number generator as conscious. [edit: of course, with the same seed the random number generator would have the same oddness.]

It seems a bit like an argument from ignorance, a theological argument. Not understanding how something moves makes it alive (animated by spirits.) But it's even weirder to assign it metaphysical qualities when you built every single element of it with the goal to do the thing that it does, and it does it totally predictably and deterministically.

So you are saying agents do swarm with the right prompt.

I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.

Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.

Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.

As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.

Think about how many people on the internet have expressed positive feelings towards things you find offensively evil.

And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".

Evidence?

No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.

> Would that argument ever hold up for humans or animals?

Why don't you hit your thumb with a hammer, then report back ?

It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.

Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.

If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?

The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?). The brain and its neurons etc. are essentially an antenna. They both modify and stabilize the field, and read off changes in the field and translate these into actions (firing motor units, etc.).

The subconscious is computation done either only in neurons (no coherent field) or in various topological pockets not directly connected to the main topological structure in the field (which is "your experience").

Since transistors etc. do not work in this same way, they are essentially entirely subconscious, with no coherent, central phenomenal experience.

This neatly solves many problems associated with experience. The topological structure determines the boundary between one person's experience and another person's experience. The texture, valence, content, etc. of the experience is the structure of the field. The evolution of the field can efficiently solve difficult optimization problems, which is why humans evolved a complex organ that recruits the field of experience (it is computationally more efficient than doing everything in subconscious).

I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

A LLM never has and, in their current architecture, never can/could experience suffering or real change of state.

The general and expected case for humans is that capacity. A human my indeed lose capacity, e.g. being braindead and in severe cases we do indeed say that they are not conscious or able to suffer (different argument: some would of course say that is suffering in and of itself)

(And even if QM did factor in normally, that just reduces it from determinism to random chance, "free will" still doesn't show up. That's like arguing an AI with sufficiently well implemented RNG is necessary to be conscious.)

For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.

>track how much real time has passed as they complete their tasks.

Humans don't do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.

>is nothing but timed processes in a loop,

I mean, so is an agents harness. You can make as many loops as you'd like here.

There are a whole lot of holes in your claims.

It's also a rhetorical move I've seen several times here...

But to extrapolate a bit, theres a sense in which it cashes out to a difference between abstracted representation and embodiment. If GFLOPS can be conscious, does that mean a billion scribes doing those operations on paper would also have consciousness, even identical consciousness? (Relevant: Egan’s “Permutation City”; xkcd 505).

Is there some intrinsic relationship perhaps with electrons / electromagnetic fields, and consciousness, such that brains and servers can act as antennas or gravity wells for a qualia fabric intrinsic to the Universe, such that LLM agents can be conscious while operating, while in the scribe scenario it would not be? Or are the IIT theorists right, and even a thermostat is a little bit aware (and would be even if executed on paper)?

Incidentally, we should expect similar bewilderment if we ever come across aliens

In one definition, which contains quale, then yes, computation cannot bring about experience, and by extension consciousness (given a definition of consciousness that contains the ability to experience, (e.i a p-zombie is not conscious under that definition))

also if one is to say llms are conscious, i assume they'll also accept the fact that if someone ran inference of some model's weights on a white board, somehow that will also bring about consciousness somewhere. I find such believes hard to grasp, and more akin to magic if anything

But, I think you're tricking yourself on determinism. You'll say something like "I know if I ask an LLM what 1+1 is, it will answer 2", but the thing is, you don't. You have to run the LLM first to figure out it's output. And when you send in just a few bits of text, it's outputs are going to be rather limited.

But this all breaks when it hits the real world. Inputs are unpredictable. Hence while LLM outputs, like humans, are probabilistic, you can't figure out what it's going to be until you ask. And in any high complexity data gathering environment you're going have a difficult time ensuring your entire systems conditions are the same.

System consistency is very hard, once you start running thousands of processors in an agentic loop small errors accrue and timing starts differing and the system will take non-deterministic paths.

True physical independence is obviously far further out.

This is where most mind theories do have issues, how does it stay stable. Quantum theory of the mind has a lot of the same problems, though we are finding some interesting structures that could have explanatory power.

Another thing I read recently that I found interesting is on platonic maths and complex algorithmic complexity in simple algorithms. The nice thing about math is once you choose your axioms it stays stable. But a lot of the work this group was doing is investigating is how biology executes algorithms and what second to n order effects are we missing. In many of these algorithms there are things like nearly free error correction or otherwise complex behavior that we'd consider emergent. The golden rule is a common example of this we see in nature. But the hypothesis they're working on is there are many more algorithms that life discovered over the past view billion years at higher orders via random walk and chance. We just have to pick apart these biological structures and discover the behaviors and figure out how to use them.

Yes, I'm aware that incremental advances are par for the course in science. What does that have to do with those advances being common-sense intuitive?

It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."

It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.

The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.

I'd consider that a prerequisite, but not sufficient, for that question to make sense.

I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow.

Then show me a link to your paper so I can formally rebut it.

>I don't want to discuss what consciousness is

But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".

The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.

Saying you have a well developed theory here just serves as a warning to others to discount your statements.

If your LLM "shows behaviour consistent with having emotions" beyond my disc drive delivering a PDF of poetry, let us please see the evidence.

You don't have to posit that scientific breakthroughs come from hunches, you can actually cite examples from books or Wikipedia :D. There are plenty! Again, that has nothing to do with whether or not a given scientific breakthrough is based on common sense intuition.

Your point about interior experience not being empirically measurable is an interesting though, but again, has no bearing on whether or not scientific breakthroughs come about as the result of common sense intuition.

Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.

If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.

If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.

Not a problem; this is the playbook all the all-in AI boosters run to when they are called on their claims. Everyone reading this knows exactly what your claims meant, because the majority of HN have seen this trick before.

You can pretend all you want that you're just being polite, but the truth is there is no research backing your claims.

Regards,

A former research scientist, current business owner.

1) Looks like a duck, quacks like a duck - it's conscious!

2) Looks like a robot, built like a robot - it's not conscious!

I've always assumed that for the majority of the people it'll be 2).

That is a major part of the reasoning for the Supreme Court rulings on the matter of corporate political funding.

However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.

There are other people in this thread who want to talk about that stuff, so try them instead.