I mean, I guess there's a lot of people that don't know the difference in what he's saying.

`rm` is a tool. You don't tell `rm` to delete things and then get annoyed when it deletes your production database because it inferred incorrectly.

It does what you tell it. Like a hammer hits what you hit with it.

`claude` can be considered a version of `rm` that can do what `rm` does (and other things - including things you don't want). Nearly everyone using it is aware of that.

An employee is not a `tool`.

`claude` is closer to an employee. You tell it to do things. It costs you money. Sometimes you get what you wanted, sometimes you get something far better, sometimes you get literally nothing, other times you get disaster.

That's why they call it an `agent`, not a `tool`.

Employers don't throw up their hands and say, no thanks, I'll do everything myself with `tool`s. They say, I'll try to learn how to better manage employees which are useful to further my goals (along with tools - because both are useful).

Not every person runs a business. Maybe in the future most people won't be using AI at all. Some people definitely will.

I'm not sure why we needed to go on this pedantic ride.

For example, a human isn't a "tool", but one serves a similar role as a tool if their job/task is well defined enough. If you're a conductor, you can view the various members of your orchestra as "tools" to produce the ideal musical output. Of course they have autonomy, they could choose to not play at all, jump into the audience and destroy their instruments, but in all likelihood they won't.

LLM's could be tools in the sense that you have a well defined task, such as "identify all facts relevant to this case from these 1000 legal documents), and 9 times out of 10 it gives you essentially what you want. That same LLM could choose to write a poem, or hack into a state government website if it wanted, but its "wants" are unpredictable and hard to specify (same as a human's) but you can still expect it to do what it should do the vast majority of the time.

That being said, the smarter an LLM gets, the more its wants are disconnected from its tasks, the "self-awareness" factor (which isn't to say LLM's are conscious, but they can portray all the verifiable elements of self-awareness, metacognition, etc), the more you have to grapple with the depth of their thinking between the input and output.

Is it sane we are speeding towards this? Who knows, maybe AI can be aligned better than humans, but if AI can't be aligned as well as humans, we can look at human history to see what happens when you can't simply order a person around like a tool and expect a certain output.

It is very difficult to disagree with anyone of the caliber such as this man, but I wanted to steer the conversation away from the generative aspect of LLMs and focus on the consumption side, because frankly, these "tools", and I think even though I might agree with Rich that LLMs by themselves are not tools, the harness I built to work with them definitely is one. I think there's an immense value there, way before you even start talking about code-generation, or code at all. I think I have failed to explain my points, because a talk we both wanted to see was about to start.

> Commenter: Well, this was a twitter-like controversial clickbait but I will say it anyway. LLM is a tool. To be more specific, it's a lossy knowledge base with a human language for querying and built-in summarization.

> Rich Hickey: Agreed. I'm talking about its embodiment in conversational systems purporting to be 'tools' for doing homework, research, coding, writing etc.

With that framing in mind, the article makes a lot of sense. AI agents are not tools for coding, they automate coding, which means you lose control, learning, predictability, etc.

In and of itself, an AI agent is a tool, can be used for things, like say making software, but if thinking of it as a tool for coding, that appears wrong, as using one changes the modality of the task, you are not coding using a tool, the coding is no longer done by you and has been automated.

As he also says, reading an AI summary of a paper is a different modality from using it to read the entirety of a paper, therefore it is not a tool for reading the way a Kindle, glasses, reader-mode are. And similar to with coding, you've lost things from automating the reading, that reading just a summary does not provide.

     In the 1960s there was recognition of some of the implications of
     human intelligence. I. J. Good wrote [10]:
     
     Let an ultraintelligent machine be defined as a machine
     that can far surpass all the intellectual activities of any
     any man however clever.  Since the design of machines is one of
     these intellectual activities, an ultraintelligent machine could
     design even better machines; there would then unquestionably
     be an "intelligence explosion," and the intelligence of man
     would be left far behind.  Thus the first ultraintelligent
     machine is the _last_ invention that man need ever make, 
     provided that the machine is docile enough to tell us how to
     keep it under control.
     ...
     It is more probable than not that, within the twentieth century,
     an ultraintelligent machine will be built and that it will be
     the last invention that man need make.
     
     Good has captured the essence of the runaway, but does not pursue
     its most disturbing consequences. Any intelligent machine of the sort
     he describes would not be humankind's "tool" -- any more than humans
     are the tools of rabbits or robins or chimpanzees. 

https://edoras.sdsu.edu/~vinge/misc/singularity.html

I’m someone that is, and has been, for most of my career, someone from the in-between lands. I don’t use paper maps from the gas station, but I have not coded much.

I basically grew up with DOS 1.0 and on. Often made a living being able to sit at a Windows workstation and think and persist so I could figure out in four hours what other people could not, in that buggy strange world of microcomputers many of us grew up in.

Those situations, now, often take me five minutes to get past, instead of four hours. It has been a game changer for me.

If you shipped something understanding none of it, you did nothing. Claude did it. Writing 'code' is not just typing characters that compile, it's maintaining a system of ideas. If you don't know anything about those underlying ideas, what did you even do? How are you not trivially replaceable?

You can argue all day about these topics, but ultimately, the paychecks, the customers, the prestige, etc will be proportional to the non-AI value that you can offer to the world.

Who says I'm a tool?

You are a tool.

- because car companies are definitely misaligned with their customers

Yes, Janet, languages do evolve. But why do you think dictionaries exists? Solely for teaching purposes? No, it is because at the end of the day, sometimes there is a need for a correct definition to establish a foundation for some argument of some kind.

But I know people are going to ignore and simply say his arguments have no validity.

A shame, beause in this he actually wrote a very clear separation of software that is a tool and aoftware that is automation. And he didn't even say AI is bad or any other thing. He simply says: AI is automation, not a tool. Use accordingly

Reminds me of Baudelaire:

"the photographic industry was the refuge of every would-be painter, every painter too ill-endowed or too lazy to complete his studies"

And they are a hell of a useful "assistant"

People try to wrangle the indeterministic outputs through deterministic layers to convert the indeterminism to something determined

"AI" as it was known until ~2023 could also be useful as an algorithmic implementation to solve an objective.

Now that definition is squishy, but still relevant.

AI is only a tool, it isn’t anything more than that. I use AI because work pays for an expensive subscription, and I’ve found it helpful… as a tool.

By his definition, we could argue that a catapult or bow and arrow are not tools or perhaps it's based on the user, many people will not find them predictable and reliable without serious practice. Likewise the same with LLM

The assumption I need to challenge is that "an ultraintelligent machine could design even better machines".

That assumes that better machines are even possible, that there's no point of bottleneck that cannot be broken, or that they are so much more intelligent than us that they too won't hit a limit pretty quickly to what they can figure out.

It also says it surpasses any man however clever, but doesn't say it can surpass the totality of man interacting and building on each other's. If you take the most clever of all man, they are not so clever as to surpass humanity's total ingenuity and invention. And it's not a given that this super intelligent machine, more clever than any one man, is able to replicate itself and work in tandem with copies of itself in a way that is as effective as how humans somehow form a dynamic system between themselves.

Lastly, and I know the premise assumes it's better at everything, including at building intelligent machines, but I think an ultra-intelligent machine could exist that struggles at building better machines. Depending on your faith, we might be the construction of lesser intelligent organisms eventually discovering and evolving into us, and yet we do not yet understand how to make a more intelligent human.