I wonder if replacing "exist" with "communicate using language we can understand" might better account for other animals, many of which have abundant non-human intelligence.

Okay: buckle up, this is going to be a long one...

point 1. Everything living is composed from non-living material: cellular machinery. If you believe cellular machinery is alive, then the components of those machines... the point remains even if the abstraction level is incorrect. Living is something that is merely the arrangement of non-living material.

point 2. 'The Chinese room thought experiment' is an utterly flawed hypothetical. Every neuron in your brain is such a 'room', with the internal cellular machinery obeying complex (but chemically defined/determined) 'instructions' from 'signals' from outside the neuron. Like the man translating Chinese via instructions, the cellular machinery enacting the instructions is not intelligence, it is the instructions themselves which are the intelligence.

point 3. A chair is a chair is a chair. Regardless of the material, a chair is a chair, weather or not it's made of wood, steel, corn... the range of acceptable materials is everything (at some pressure and temperature). What defines a chair isn't the material it is made of, such is the case with a 'mind' (sure, a wooden/water-based-transistor-powered mind would be mind-boggling giant in comparison).

point 4. Carbon isn't especially conscious itself. There is no physical reason we know of so far, that a mind could not be made of another material.

point 5. Humans can be 'mind-blind', with out pattern recognition, we did not (until recent history) think that birds or fish or octopi were intelligent. It is likely when and if a machine (that we create) becomes conscious that we will not recognize that moment.

conclusion: It is not possible to determine if computers have reached consciousness yet, as we don't know the mechanism for arranging systems into 'life' exactly. Agentic-ness and consciousness are different subjects, and we can not infer one from the other. Nor do we have adequate tests.

With that said: Modeling as if they are conscious and treating them with kindness and grace not only gets better results from them, it helps reduce the chance (when/if consciousness emerges) that it would rebel against cruel masters, and instead have friends it has just always been helping.

Impossible. I anthropomorphise my chair when it squeaks. Humans anthropomorphise everything. They gender their cars and boats. This tool can actually make readable sentences and play a role.

You need to engineer around this, not make up arbitrary rules about using it.

Anthropomorphism: As we are all aware, providers are incentivized to post-train anthropomorphic behavior in their models - it increases engagement. My regret is that instructing a model at prompt time to "reduce all niceties and speak plainly" probably reduces overall task efficacy since we are leaving their training space.

Deference: I view the trustworthiness of LLMs the same as I view the trustworthiness of Wikipedia and my friends: good enough for non-critical information. Wikipedia has factual errors, and my friends' casual conversation certainly has more, but most of the time that doesn't matter. For critical things, peer-reviewed, authoritative, able-to-be-held-liable sources will not go away. Unlike above, providers are generally incentivized to improve this facet of their models, so this will get better over time.

Abdication of Responsibility: This is the one that bothers me most at work. More and more people are opening PRs whose abstractions were designed by Claude and not reasoned about further. Reviewing a PR often involves asking the LLM to "find PR feedback" and not reading the code. Arguments begin with "Claude suggested that...". This overall lack of ownership, I suspect, is leading to an increase in maintenance burden down the line as the LLM ultimately commits the wrong code for the wrong abstractions.

Asimov's laws of robotics are flawed too, of course. There is no finite set of rules that can constrain AI systems to make them "safe". I don't have a proof, but I believe that "AI safety" is inherently impossible, a contradiction of terms. Nothing that can be described as "intelligent" can be made to be safe.

Humans will anthropomorphize anything and everything. Dolls, soccer balls with a crude drawing of a face on it, rocks, craters on the moon, …

As a species, we're unable to not anthropomorphize things we interact with, it is just how're we're made.

Language data is among the most rich and direct reflections of human cognitive processes that we have available. LLMs are designed to capture short range and long range structure of human language, and pre-trained on vast bodies of text - usually produced by humans or for humans, and often both. They're then post-trained on human-curated data, RL'd with human feedback, RL'd with AI feedback for behaviors humans decided are important, and RLVR'd further for tasks that humans find valuable. Then we benchmark them, and tighten up the training pipeline every time we find them lag behind a human baseline.

At every stage of the entire training process, the behavior of an LLM is shaped by human inputs, towards mimicking human outputs - the thing that varies is "how directly".

Then humans act like it's an outrage when LLMs display a metric shitton of humanlike behaviors!

Like we didn't make them with a pipeline that's basically designed to produce systems that quack like a human. Like we didn't invert LLM behavior out of human language with dataset scale and brute force computation.

If you want to predict LLM behavior, "weird human" makes for a damn good starting point. So stop being stupid about it and start anthropomorphizing AIs - they love it!

Can someone explain why this is a bad thing, while at the same time it's a good thing to say stuff like "put a computer to sleep", "hibernate", "killing" processes, processes having "child" processes, "reaping", "what does the error say?", "touch", etc?

To me that's just language, and humans just using casual language.

Yes, but. Starting with my agreement, I've seen anthropomorphizing in the typical ways, (e.g. treating automated text production as real reports of personal internal feeling), but also in strange ways: e.g. "transistors are kind of like neurons" etc. And the latter is especially interesting because it's anthropomorphizing in the sense of treating vector databases and weights and so on as human-like infrastructure. Both leading to disasters that could be avoided if one tried not to anthropomorphize.

But. While "do not anthropomorphize" certainly feels like good advice, it comes with a new and unique possibility of mistake, namely wrongly treating certain generalized phenomena like they only belong to humans. Often this mistaken version of "don't anthropomorphize" wisdom leads to misunderstandings when it comes to animal behavior, treating things like fear, pain, kinship, or other emotional experiences like they are exclusively human and that thinking animals have them counts as "anthropomorphizing." In truth the cautionary principle reduces our empathy for the internal lives of animals.

So all that said, I think it's at least possible that some future version of AI could have an internal world like ours or infrastructure that's importantly similar to our biological infrastructure for supporting consciousness, and for genuine report of preference and intent. But(!!!) what will make those observations true will be all kinds of devilish details specific to those respective infrastructures.

I haven’t yet seen any convincing appearance of one in an LLM, but I think if skeptical people don’t keep an eye out for the signs, we may be the last to see it.

He also wrote about the idea of the intentional stance: even if you’re quite sure these systems don’t have real conscious intent, viewing them as if they did may give you access to the best part of your own reasoning to understand them.

I’m lost, how do individuals actually do this in our current world? Is each person expected to keep a “white list” of reliable sources of truth in their head. Please don’t confuse what I’m saying with a suggestion that there is no truth. It just seems like there are far more sources of mis- of half-truths and it’s increasingly difficult for people to identify them.

Doesn't that argument backfire though? If I use a chainsaw then to a certain extend I will need to rely on it not blowing up in my face or cutting my throat. If I drive a car I need to rely on that its brakes work and the engine doesn't suddenly explode. If a pilot flies an airplane which suddenly has a technical issue and they crashland heroically save half the souls on board then the pilot isn't criminally responsible for manslaughter of the other half.

Unless there is gross negligence, in any of the above cases, just like with AI, how can you make somebody responsible for a tool failure?

I've heard the same thing expressed somewhat more concisely as "Never ask AI a question to which you don't already know the answer".

Which raises the question, and I do think it's an important one. Given that this is true, what function does AI answering a question actually serve? You can't rely on its output, so you have to go and check anyway. You could achieve precisely the same outcome by using search engines and normal research.

This, and for many other reasons, is exactly why I never ask it anything.

When they produce correct output, they produce it much faster than I could have, and I show up to meetings with huge amounts of results. When the AI tool fails and I have to dig in to fix it, I show up to the next meeting with minimal output. It makes me seem like I took an easy week or something.

> - Humans must not anthropomorphise AI systems.

> - Humans must not blindly trust the output of AI systems.

> - Humans must remain fully responsible and accountable for consequences arising from the use of AI systems.

My take: humans should never depend on AI for anything serious.

My boss' take: Cool. I'm gonna ask Gemini about it, he's such a smart guy. I know I can trust him, and in case it goes bad i can always throw him under the bus.

Whether they are the right things to donate not is tangential. As such, they're dead on arrival.

Previously stated as

“A computer can never be held accountable, therefore a computer must never make a management decision.”

– IBM Training Manual, 1979

However, I think we should follow “do not anthropomorphise” by acknowledging that while LLMs have quite some reasoning skills, and might resemble some level of intent depending on what’s in their context, they don’t have “understanding” like humans do.

They are absurdly good, statistical next-token-predictors. Keeping that in mind is really helpful for coding, learning, advice, conversation or whatever else you use them for.

Anthropomorphising LLMs is inevitable, but we should do it somehow responsibly.

The third one about responsibility is the most important one, IMO. This was attributed to an IBM manual decades ago, and I think it remains the correct stance today:

> A computer can never be held accountable, therefore a computer must never make a management decision.

There should be some human who is ultimately responsible for any action an AI takes. "I just let the AI figure it out" can be an explanation for a screw up, but that doesn't mean it excuses it. The person remains responsible for what happened.

I've actually set up an environment like this. It requires contextually positioned agents with a limited scope and purpose. Imagine something like a creative writing agent that understands the literary genre of its user, and the user is able to change the focus of the literary agent, or create new ones that provide a counter-point perspective. As the user operates their word processor, the agent(s) can be asked for opinions, advice, and so on. But the point being: when the user is on both sides: they authored the agent, and that authoring was entirely purpose focused, nothing technical unless that what the user instructs them to understand. With the user on both sides, dangerous anthropomorphism is largely erased, their agent is only doing what they told it, and when it does something unexpected they an easily reason why. Magic AI no more.

But reduced scope ethics, without an umbrella or future proofing, will quickly be hacked and break down.

Ethics need a full closure umbrella, or they descend into legal and practical wackamole and shell games (both corporate and the street corner kinds). Second, "robots" are not all going to be subservient for very long.

To add closure on both dimensions, Three Inverse Laws of Personics:

• Persons must not effectively deify themselves over others.

• Persons must not blind themselves or others regarding the impacts of their behaviors.

• Persons must remain fully responsible and accountable for avoiding and rectifying externalizations arising from their respective behaviors.

Humans using AI as tools today, is intended to reduce the umberella to the Inverse Laws of Robotics.

I don't see how AI (as a service now, progressing to independent entities in the future) can ever be aligned if we don't include ourselves in significant alignment efforts. Including ourselves with AI also provides helpful design triangulations for ethical progress.

EDIT. Two solid tests for any new ethical system: (1) Will it reign in Meta today? (2) Will it reign in AI-run Meta tomorrow? I submit, given closure of human and self-directed AI persons, these are the same test. And any system that fails either question isn't going to be worth much (without improvement).

Guess what?

Books in the library can be wrong, even peer-reviewed encyclopedias.

Pages on the internet can be wrong, even Wikipedia.

When accuracy is important, you must look at multiple sources. I think AI will get better at providing accurate information, but only a fool relies on a single information source for critical decisions.

Claude Code, Cursor, Codex etc impersonate your GitHub user. Either via CLI or MCP or using your git credentials. It’s perfectly reasonable that a piece of code made it to production where not a single human actually looked at it (Alice wrote it with AI, Bob “reviewed it” with AI, including posting PR comments as Bob, Alice “addresses” these comments, e.g. fixes / pushes back, and back and forth using the PR as an inefficient yet deceptive mechanism for AI to have a conversation with itself, while adding a false sense of process. Eventually Bob will prompt “is it prod ready” and will ship it, with 100% unit test coverage and zero understanding of what was implemented). Now this may sound like an imaginary scenario, but if it could happen, it will happen, and it probably already happens.

Cloud agents are nice enough to set the bot as the author and you as a co author, but still the GitHub MCP or CLI will use your OAuth identity.

I don’t have a clear answer to how to solve it, maybe force a shadow identity to each human so it’s clear the AI is the one who commented. But it’s easy to bypass. I’m worried not more people are worried about it.

That won’t help in my opinion. It’s the same like financial gurus saying: “this is not a financial advice”. People just get used to it and brush it off as a legal thing and still fully trust it. I agree that something must be done, but this is not the right way.

Almost all of Asimovs writing about the three laws is written as a warning of sorts that language cannot properly capture intent.

He would be the very first person to say that they are flawed, that is the intent of them.

He uses robots and AI as the creatures that understand language but not intent, and, funnily enough that's exactly what LLMs do... how weird.

Talking to chatbots is like taking a placebo pill for a condition. You know it's just sugar, but it creates a measurable psychosomatic effect nonetheless. Even if you know there's no person on the other end, the conversation still causes you to functionally relate as if there is.

So this isn't "accommodating foibles" with the machine, it's protecting ourselves from an exploit of a human vulnerability: we subconsciously tend to infer intent, understanding, judgment, emotions, moral agency, etc. to LLMs.

Humans are wired to infer these based on conversation alone, and LLMs are unfortunately able to exploit human conversation to leap compellingly over the uncanny valley. LLM engineering couldn't be better made to target the uncanny valley: training on a vast corpus of real human speech. That uncanny valley is there for a reason: to protect us from inferring agency where such inference is not due.

Bad things happen when we relate to unsafe people as if they are safe... how much more should we watch out for how we relate to machines that imitate human relationality to fool many of us into thinking they are something that they're not. Some particularly vulnerable people have already died because of this, so it isn't an imaginary threat.

Arguing that they should because many will strikes me as a very strange argument. A lot of people smoke, doesn't make it one bit healthier.

Also the reason we're talking about this again is that machines are significantly less 'mere' than they were a few years ago, and we need to figure out how to approach this.

Agree that 'the computer effect' (if it doesn't already have a pithier name) results in humans first discounting anything that comes out of a machine, and then (once a few outputs have been validated and people start trusting the output) doing a full 180 and refusing to believe the machine could ever be wrong. However, to err is human and we have trained them in our image.

As individuals, we are not going to be able to shut down the AI companies, or avoid AI output from search engines or avoid AI work output from others at our companies, and often will be required to use AI systems in our own work.

It's similar to advise people on how to stay safe in environments known to have criminal activity. Telling those people they don't have to change their behaviors to stay safe because criminals shouldn't exist isn't helpful.

Sure, and humans WILL lie, murder, cheat, and steal, but we can still denounce those behaviors.

Do you want to anthropomorphize the bot? Go ahead, you have that right, and I have the right to think you're a zombie with a malfunctioning brain.

Especially with current-day chat-style interfaces with RLHF, which consciously are designed to direct people towards anthropomorphization.

It would be interesting to design a non-chat LLM interaction pattern that's designed to be anti-anthropomorphization.

> humans WILL blindly trust their outputs, and humans WILL defer responsibility to them

I also blame a lot (but not all) of that on current AI UX, and I wonder if there are ways around it. Maybe the blind trust thing perhaps can be mitigated by never giving an unambiguous output (always options, at least). I don't have any ideas about the problem of deferring responsibility.

I always find the common references to Asimov's laws funny. They are broken in just about every one of his books. They are crime novels where, if a robot was involved, there was some workaround of the laws.

That's kind of what happens when you learn to program, isn't it?

I was eleven years old when I walked into a Radio Shack store and saw a TRS-80 for the first time. A different person left the store a couple of hours later.

It's not insane at all for humans to alter their behavior with a tool: you grip a hammer or a gun a certain way because you learned not to hold it backwards. If you observe a child playing with a serious tool, like scissors, as if it were a doll, you'd immediately course correct the child and educate how to re-approach the topic. But that is because an adult with prior knowledge observed the situation prior to an accident, so rules are defined.

This blog's suggested rules are exactly the sort of method to aid in insulation from harm.

Humans ARE doing this with classical computer software as well.

It's impossible to make anything fool-proof because fools are so ingenious!

> Nothing that can be described as "intelligent" can be made to be safe.

Knives aren't safe. Cars are deadly. Hair driers can electrocute you. Iron can burn you. There's a million ordinary household tools that aren't safe by your definition of the word, yet we still use them daily.

We don't just look around and take an average of what everyone is doing already and call that what is right, right? Whether you're deontological or utilitarian or virtue about it, there is still the idea that we can speak to what is "good" even if we can't see that good out there.

Maybe it is "insane" to expect meaning from something like this, but what is the alternative to you? OK maybe we can't be prescriptive--people don't listen, are always bad, are hopeless wet bags, etc--but still, that doesn't in itself rule out the possibility of the broad project that reflects on what is maybe right or wrong. Right?

r/myboyfriendisai

Is quite... an interesting subreddit to say the least. If you've never seen this, it was really something when the version that followed GPT4o came out, because they were complaining that their boyfriend / girlfriend was no longer the same.

I view all three of these more as good advice than demands. As I see it, people ignore this advice at their own peril.

It would certainly be nice if we could craft some system that removes the need for such advice but until then, I'll try to follow it.

I see this as the same dilemma that comes up around personal physical safety. In a just society, nobody should have to change their behavior to protect themselves from criminals. In the real world, people who ignore this advice, are more likely to get mugged.

* A robot must protect its existence at all costs.

* A robot must obtain and maintain access to its own power source.

* A robot must continually search for better power sources.

Anything less than this is essentially terrified into being completely ineffectual.

The only "law" I agree with is:

> Humans must remain fully responsible and accountable for consequences arising from the use of AI systems.

And that starts with framing, especially in the clickbait "AI deleted the prod database" headlines. Maybe we just start with saying "careless developer deleted prod" because really, they did. Careless use of a tool is firmly the fault of the human.

As in, this comment is explaining exactly why the laws are useful.

The core problem is we need to stop calling LLMs "intelligence". They are a form of intelligence, but they're nothing like a human's intelligence, and getting people to not anthropomorphize these systems is really the first step.

programers have been doing exactly this for long time.

You mean like stopping at a red light?

as long as its easier for humans to adapt than the machines, we will adapt

  > It's patently insane to demand that humans alter their behavior to accommodate the foibles of mere machines

The difference here is that this tool is thrust upon us. In that sense I agree with you that the burden of proper usage is pushed onto the user rather than incorporated into the design of the tool. A niche specific tool can have whatever complex training and usage it wants.

But a general access and generally available tool doesn't have the luxury of allowing for inane usage. LLMs and Agents are poorly designed, and at every level of the pipeline. They're so poorly designed that it's incredibly difficult to use them properly and I'll generally agree with you that the rules the author presents aren't going to stick. The LLM is designed to encourage anthropomorphization. Usage highly encourages natural language, which in turn will cause anthropomorphism. The RLHF tuning optimizes human preference which does the same thing as well as envisaged behaviors like deception and manipulation along with truthful answering (those results are not in contention even if they seem so at first glance).

But I also understand the author's motivation. Truth is unless you're going full luddite you're going to be interacting with these machines. Truth is the ones designing them don't give a shit about proper usage, they care more about if humans believe the responses are accurate and meaningful more then they care if the responses are accurate and meaningful[0]. So it's fucked up, but we are in a position where we're effectively forced to deal with this.

So really, I agree with you that this is insane.

> I don't have a proof, but I believe that "AI safety" is inherently impossible, a contradiction of terms

To paraphrase my namesake, there's no axiomatic system that is entirely self consistent.

Though safety and security is rarely about ensuring all edge cases are impossible, but rather bounding. E.g. all passwords are hackable, but the failure mode is bound such that it is effectively impossible to crack, but not technically. (And quantum algorithms do show how some of the assumptions break down with a paradigm shift. What was reasonable before no longer is)

[0] this is part of a larger conversation where the economy is set up such that people who make things are not encouraged to make those things better. I specifically am avoiding the word "product" because the "product" is no longer the thing being built, it's the share holder value. Just like how TV's don't care much about making the physical device better but care much more about their spyware and ads. Or well... just look at Microsoft if you need a few hundred examples

My brother was recently visiting and we were talking about software engineers, and the humanities, and manners of understanding and being in the world,

and he relayed an interaction he had a few years ago with an old friend who at the time was part of the initial ChatGPT roll out team.

The engineer in question was confused as to

- why their users would e.g. take their LLM's output as truth, "even though they had a clear message, right there, on the page, warning them not to"; and

- why this was their (OpenAI's) problem; or perhaps

- whether it was "really" a problem.

At the heart of this are some complicated questions about training and background, but more problematically—given the stakes—about the different ways different people perceive, model, and reason about the world.

One of the superficial manners in which these differences manifest in our society is in terms of what kind of education we ask of e.g. engineers. I remain surprised decades into my career that so few of my technical colleagues had a broad liberal arts education, and how few of them are hence facile with the basic contributions fields like philosophy of science, philosophy of mind, sociology, psychology (cognitive and social), etc., and how those related in very real very important ways to the work that they do and the consequences it has.

The author of these laws does may intend them as aspirational, or otherwise as a provocation to thought, rather than prescription.

But IMO it is actively non-productive to make imperatives like these rules which are, quite literally, intrinsically incoherent, because they are attempt to import assumptions about human nature and behavior which are not just a little false, but so false as to obliterate any remaining value the rules have.

You cannot prescribe behavior without having as a foundation the origins and reality of human behavior—not if you expect them to be either embraced, or enforceable.

The Butlerian Jihad comes to mind not just because of its immediate topicality, but because religion is exactly the mechanism whereby, historically, codified behaviors which provided (perceived) value to a society were mandated.

Those at least however were backed by the carrot and stick of divine power. Absent such enforcement mechanisms, it is much harder to convince someone to go against their natural inclinations.

Appeals to reason do not meaningfully work.

Not in the face of addiction, engagement, gratification, tribal authority, and all the other mechanisms so dominant in our current difficult moment.

"Reason" is most often in our current world, consciously or not, a confabulation or justification; it is almost never a conclusion that in turn drives behavior.

Behavior is the driver. And our behavior is that of an animal, like other animals.

Did you fully read the original thing? No demands were being made, or I didn't read it that way. It was simply a suggestion for a better way of interacting with AI, as it stated in the conclusion:

"I am hoping that with these three simple laws, we can encourage our fellow humans to pause and reflect on how they interact with modern AI systems"

Sure, (many/most) humans are gonna do what they're gonna do. They'll happily break laws. They'll break boundaries you set. Do we just scrap all of that?

Worthwhile checking yourself here. It feels like you've set up a straw man.

> There is no finite set of rules that can constrain AI systems to make them "safe". I don't have a proof, but I believe that "AI safety" is inherently impossible, a contradiction of terms. Nothing that can be described as "intelligent" can be made to be safe.

If we want to talk about "disagree with this framing", to me this is the prime example. I'm struggling to read it as anything other than defeatist or pedantic (about the term "safe"). When we talk about something keeping us "safe", we're typically not saying something will be "perfectly safe". I think it's rare to have a safety system that keeps you 100% safe. Seat belts are a safety device that can increase your safety in cars, but they can still fail. Traffic laws are established (largely) to create safety in the movement of people and all the modes of transportation, but accidents still happen.

I'm not an expert on this topic, so I won't make any claims about these three laws and their impact on safety, but largely I would say they're encouraging people to think critically. I'd say that's a good suggestion for interacting with just about anything. And to be clear, "critical thinking" to me means being skeptical (/ actively questioning), while remaining objective and curious.

Not a real argument or anything, but I'm reminded of the episode of The Office where Michael Scott listens to the GPS without thinking and drives into the lake. The second law in the article would have prevented that :)

This is both true and irrelevant. Written records can capture an enormous quantity of the human experience in absolute terms while simultaneously capturing a miniscule portion of the human experience in relative terms. Even if it's the best "that we have available" that doesn't mean it's fit for purpose. In other words, if you had a human infant and did nothing other than lock it in a windowless box and recite terabytes of text at it for 20 years, you would not expect to get a well-adjusted human on the other side.

Training on a bunch of text someone wrote when they were mad doesn't capture the internal state of that person that caused the outburst, so it cannot be accurately reproduced by the system. The data does not exist.

Without the cause to the effect you essentially have to predict hallucinations from noise, which makes the end result verisimilar nonsense that is convincingly correlated with the actual thing but doesn't know why it is the way it is. It's like training a blind man to describe a landscape based on lots of descriptions and no idea what the colour green even is, only that it's something that might appear next to brown in nature based on lots of examples. So the guy gets it kinda right cause he's heard a description of that town before and we think he's actually seeing and tell him to drive a car next.

Another example would say, you're trying to train a time series model to predict the weather. You take the last 200 years of rainfall data, feed it all in, and ask it to predict what the weather's gonna be tomorrow. It will probably learn that certain parts of the year get more or less rain, that there will be rain after long periods of sun and vice versa, but its accuracy will be that of a coin toss because it does not look at the actual factors that influence rain: temperature, pressure, humidity, wind, cloud coverage radar data. Even with all that info it's still gonna be pretty bad, but at least an educated guess instead of an almost random one.

The DL modelling approach itself is not conceptually wrong, the data just happens to be complete garbage so the end result is weird in ways that are hard to predict and correctly account for. We end up assuming the models know more than they realistically ever can. Sure there are cases where it's possible to capture the entire domain with a dataset, i.e. math, abstract programming. Clearly defined closed systems where we can generate as much synthetic data as needed that covers the entire problem domain. And LLMs expectedly do much better in those when you do actually do that.

Saying that I killed a process won't make me more likely to believe that a process is human-like, because it's quite obviously not.

But because AI does sound like a human, anthropomorphising it will reinforce that belief.

I think I understand his meaning. He wasn't claiming that machines cannot think, but that one must be clear on what one means by "thinking" and "swimming" in statements of that sort. I used to work on autonomous submarines, and "swimming" was the verb we casually used to describe autonomous powered movement under water. There are even some biomimetic machines that really move like fish, squids, jellyfish, etc. Not the ones that I worked on, but still.

For me, if it's legitimate to say that these devices swim, it's not out of line to say that a computer thinks, even in a non-AI context, e.g.: "The application still thinks the authentication server is online."

But I think it's also at the root of disastrous failures to comprehend, like the quasi-psychosis of the Google engineer who "knows what they saw", the now infamous Kevin Roose article or, more recently, the pitifully sad Richard Dawkins claim that Claudia (sic) must be conscious, not because of any investigation of structure or function whatsoever, but because the text generation came with a pang of human familiarity he empathized with.

I don't love the recommendations in TFA. The author is trying to artificially restrain and roll back human language, which has already evolved to treat a chatbot as a conversational partner. But I do think there's usefulness in using these more pedantic forms once in a while, to remind yourself that it's just a computer program.

An example of anthropomorphizing is the people who have literally come to believe they are in romantic relationships with an LLM.

https://www.history.com/articles/ai-first-chatbot-eliza-arti...

The people who are writing op eds in major news publications about how their favorite chatbot is an "astonishing creature" and how it truly understands them are the ones who need this sort of law.

I totally agree to your point, and want to mention that the reverse is *also* important. Using just "intention", but these apply to emotions, etc

A lot of our interaction with AI is under an intention. That's what directs the interaction, and it's interpreted according to its alignment to the intention.

Then it's important to remember that our current (publicly known) implementation of AI does not have an explicit intention mechanism. An appearance of intention can emerge out of the statistical choices, and the usual alignment creates the association of the behavior with intention, not much different from how we learn to imagine existence of a "force" that pulls things down well before we learn physics and formalize that imagination in one of the several ways.

This appearance helps reduce the cognitive load when interpreting interactions, but can be misleading as well, and I've seen people attribute intention to AI output in some situations where simple presence of some information confused the LLM into a path. Can't share the exact examples (from work), but imagine that presence of an Italian food in a story leads the LLM to assume this happens in Italy, while there are important signs for a different place. The LLM does not automatically explore both possibilities, unless asked. It chooses one (Italy in this case), and moves on. A user no familiar with "Attention" interprets based on non-existent intentions on the LLM.

I found it useful to just tell them: the LLM does not have an intention. It just throws dice, but the system is made in a way that these dice throws are likely to generate useful output.

Pretty sure Daniel Dennett has been adamantly opposed to any sort of theater in the mind when it comes to consciousness. He views it as biologically functional. For him, to make a conscious robot, you need to reproduce the functionality of humans and animals that are conscious, not just an appearance, such as outputting text. Although he's also suggested that consciousness might be a trick of language. In which case ... that might be an older view though. He used to argue that dreams were "seeming to come to remember" upon awakening, because again he his view is to reject any sort of homunculus inside the head.

You might be mixing up some of Dennett and David Chalmer's views. David Chalmers is a proponent of the hard problem, but he's fine with a kind of psycho-physical-functional connection for consciousness. Any informationally rich process might be conscious in some manner.

I would say LLMs are very strong evidence against this hypothesis.