The problem with LLMs isn't hallucination, it's context specific confidence

The problem with LLMs isn't hallucination, it's context specific confidence(signalfire.com)

4 points by kerwioru9238492 260 days ago | 4 comments

_wire_ 260 days ago |

"The problem with Magic 8-ball is lack of context specific confidence in its answers"

This article and attendant comments reveal the AI sector is turning to co-dependent excuse making for a technology that clearly can't live up to its hype.

Get ready for phrenology of AI...

"I am going to need to visit your data center to lay hands on the subject."

zviugfd 260 days ago |

It feels like most safety work is turning LLMs into overly cautious assistants and I like how this points out that we could be trading away imagination for a false sense of reliability.

alganet 260 days ago |

> Humans don’t get rewarded for saying “I don’t know” to every question, because that’s not useful.

Humans get rewarded for thinking "I don't know", a lot. That's why it's hard to compare.

> A model that always bluffs

A model doesn't bluff. It feels to us humans that they bluff, but there is no bluff mechanics in play. The model doesn't assess the prompter's ability to call their bluff. It's not hiding that it doesn't know something. It's just not reached a predictable point in a sequence of token predictions that can or not have something that resembles a call to what resembles a bluff.

Up to the point it's corrected, the model's representation of what was asked is the best it can do. It has no means to judge itself. Which leads to...

> The real issue isn’t that models make things up; it’s that they don’t clearly signal how confident they are when they do.

Which sounds like exactly what I said, but it's not. Signaling confidence is just a more convincing faux-bluff. Signaling is a side-effect of bluffing, a symptom, not the real thing (which is more related to asessing whoever is on the other side of the conversation).

> Imagining things, seeing problems from the wrong angle, and even fabricating explanations are the seeds of creativity.

I agree with this. However, Newton was not bluffing, he was right and confident about it, and right about being confident about it. It just turns out that his description was of a lesser knowledge resolution than Einsten's.

For this to work, we need lots of "connective tissue" ideas. Roads we can explore freely without being called liars. Things we can say without saying that these things are true or false, without the need for being confident or right, without being assessed directly. This is outside the realm of bluffing or saying useful things. It's quite the opposite.

When people saw comets and described them as dragons in the sky, they were not hallucinating or telling lies, they were preserving some connective tissue idea the best they could, outside of the realm of being right or wrong. This were not bluffs. There were some "truths" about their mistakes, or something useful (they were unadvertedly recording astronomical data, before astronomy existed). Those humans felt that was important, those stories stuck. Can we say the same thing about LLM hallucinations? I don't think we're ready to answer that.

So, yes. Hallucinations could be a feature, but there's a lot missing here.

kerwioru9238492 259 days ago | |

One issue right now is that in a lot of ML benchmarks models get rewarded for guessing multiple choice questions due to the probability of being right. In addition to that, people have tuned models via RLHF to be very confident because people think confident responses sound good. These two paired together resembles bluffing because models will guess at answers very confidently rather than saying "I don't know".