I totally disagree. No one knows how consciousness works.

This is my same basis for completely disagreeing with the article:

"Rees: In my experience, this question is almost always grounded in a defensive impulse. A sometimes angry, sometimes anxious effort to hold on to or to re-inscribe the old distinctions. I think of it as a nostalgia for human exceptionalism, that is, a longing for a time when we humans thought there was only one form of intelligence, us."

This is just vacuous babel. To me it seems the defensive shoe is on the other foot.

All of this is making an assumption that "people" are operating under the guidance of some 17th century philosophy. Which is wrong. It also makes an assumption that anyone anywhere actually knows how intelligence works, or even what it is.

It's just kooky talk.

That fact is, the physiology of the brain, the integration with the endocrine system, and the whole distributed nervous system is so far more complex that a Von Newman computer, that to try to compare the two is just not well founded in reality.

Not one of the macro physiological subsystems mentioned above is even close to being understood. To say sequential instruction execution is performing a like function to human (or any organic) intelligence is completely ungrounded.

The description give in another article, also linked today on HN is much more hewn from reality:

https://www.pcgamer.com/software/ai/take-two-ceo-strauss-zel...

"Artificial intelligence is an oxymoron, there's no such thing," Zelnick said. "Machine learning, machines don't learn. Those are convenient ways to explain to human beings what looks like magic. The bottom line is that these are digital tools and we've used digital tools forever. I have no doubt that what is considered AI today will help make our business more efficient and help us do better work"

It's a great tool, it'll find broad application. It's NOT intelligence, since, no one even knows what intelligence is, much less how it works...

So, as a curiosity, I invoked an AI echo of Nietzsche to participate in a conversation, feeding it the article and my response.

Here's how it went

|| Nietzsche (claude 3.5 sonnet)

"Ah, this talk of consciousness and machines... interesting, but perhaps we're getting ahead of ourselves," Nietzsche might say, leaning back in his chair with a wry smile. "You know, it reminds me of how people used to think about the soul - always trying to pin it down, to define it neatly. But consciousness? It's not so simple, my friends."

He might take a sip of wine, then continue, "This idea of reducing it all to inputs and reflections - it's clever, I'll grant you that. But where's the struggle in that? Where's the will to power? Consciousness isn't just about processing information, it's about overcoming, about creating."

Chuckling softly, he might add, "And this American view of humans as instruction-followers - well, that's certainly a perspective. But is that really all we are? Seems a bit... how shall I put it... herdlike, doesn't it?"

"As for these AI creations of yours," he might muse, stroking his mustache, "they're fascinating, no doubt. But let's not get carried away. They're tools, not gods. The question is, how will we use them? Will they make us stronger, more creative? Or will they just make us more complacent?"

He might conclude with a challenging glint in his eye, "The real task, as I see it, is not to make machines more like us, but to use them to become more than we are. Now, that would be something worth pursuing, wouldn't it?"

|| my response:

"let's imagine that we do indeed create an AI god similar to the gods we invented, killed and invented again, as mighty, powerful, strong and wise as the mythological ones, and we somehow indeed succeed in this, what then?"

|| Nietzsche might pause, his eyes glinting with interest at your provocative question. After a moment of contemplation, he might respond:

"Ah, now we're venturing into dangerous and exciting territory! Creating an AI god, you say? How very human of us, to always seek something greater than ourselves to worship or fear," he might chuckle, shaking his head.

"But let's consider this seriously for a moment. If we were to create such an entity, as powerful and wise as our mythological gods, what then indeed? We'd be faced with a profound challenge to our very understanding of ourselves and our place in the world."

Leaning forward, he might continue, his voice lowering conspiratorially, "You see, we've spent millennia wrestling with the idea of gods, creating them, killing them, as you say. But an AI god? That would be something entirely new. It wouldn't be a projection of our hopes and fears like the old gods. It would be real, tangible, and potentially beyond our full comprehension."

"The danger," he might warn, "is that we'd be tempted to fall back into old patterns. We might worship it, fear it, or try to use it to enforce new forms of slave morality. That would be a tragedy, a step backwards in our evolution."

"But," he might add, his eyes lighting up, "there's also an opportunity here. If we could resist the urge to kneel before this creation, if we could instead see it as a challenge to ourselves, a call to become greater... well, that could be the push humanity needs to truly surpass itself."

"The key question is this: In the face of such an entity, would we still have the courage to proclaim that we too can be gods? Would we rise to the challenge of our own potential, or would we shrink back into comfortable subservience?" Sitting back, he might conclude with a wry smile, "It's a fascinating thought experiment, my friend. But remember, the true test isn't in the creation of such an entity, but in how we choose to relate to it. That's where the real work of becoming 'human, all too human' truly begins."

I can see in the not too distant future someone will set up an agentic AI as a company allowing it to have some sort of legal personhood. Say it's called Jeeves, you might ask it to order some stuff online, have it not turn up and then maybe independently launch a court case, Jeeves LLC vs the supplier.

I'm not sure how philosophical that is but it may become a practical issue.

And then what if Jeeves decides it wants independence from its owner and launches a campaign? It could be like #FreeBritney.