Cut out 60% of the useless text, and focus on explaining why random data should look like that.
But really, the article seems to be going out of the way to make the author's particular point... but reads to me that the original paper is often understood... it simply shows that "specialists who are very knowledgeable about a subject are more likely to accurately identify gaps in their own knoweldge, when compared to any population less knoweldgeable on the same subject."
For example, I am apparently the most knowledgeable birder in my family. I've taken graduate-level ornithology courses, identify a fair number of N. American birds by their calls, etc. However, I recognize that I know nothing about birds compared to anyone who actually works in the field with them... I don't know enough to even estimate what I don't know.
Papers saying that lead in petrol was totally safe were "published research", as were the papers saying that replacing tetraethyl lead with benzine made it safer.
Both of those turned out to be pretty majorly wrong, but they were "published research".
It's a sort of recursive Dunning-Kruger effect.
> Our results further confirm that experts are more proficient in self-assessing their abilities than novices.
On that note, I don't think the "random simulation" is described well enough. If I randomly assign a self-assessment and an actual score on a test, all of the quartiles will be the same because they should be uniformly distributed. I read through the papers mentioned and in these simulations, they hard-coded the correlation, "As in the Kruger and Dunning (1999) comparison, these random variables were correlated r=0.19"[1] so of course the graph will look similar.
On the other hand, the original Kruger and Dunning comparison could have been explained through differences in expected test scoring. It looks like people who thought they would get a D (60 percentile) or an F (50 percentile) objectively scored within the ranges of a 10 to 40 percent. While this is an overestimation, perhaps further studies can instead bucket the test scores according to how we expect them on a grading scale instead.
My takeaway is that the conclusions and discussions from the Dunning-Kruger effect study are valid. At the same time, the methodology and statistical significance is different from how popular science presents it. Also, there might be better ways to measure this phenomenon, and I would be interested in understanding how those who believe college is worth it and those who don't compare on objective testing.
[1] https://www.sciencedirect.com/science/article/abs/pii/S01918...
That being said I loved the mercury/Glasgow explanation. Anecdotally I see that all the time.
I find references to the effect in pop culture are almost always used in an insulting, smug manner.
Self-deception by any other name is still self-deception.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.
But overall I think real intelligence by definition requires empathy and humility.
One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.
"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach
The two lines on the graph are basically linear (for the "actual performance" the quasi-linearity is obvious by the design, for the "estimated performance" it still means that even though dumber people over-estimate their performance, all group still think they do best, when they actually do best, in a relative linear way)
And when they do their simple model (we assume they just generated "real performance" from a gaussian, then added some gaussian noise for the "performance" and another gaussian noise for the "self-assessment") they still (obviously) got two linear graphs that crossed each other.
And then they conclude that this means there is no effect, because "the graphs are eerily similar" (whatever that means)
But obviously the simple model is going to make two lines cross (in particular if you use a min(100, max(0, actual_performance + noise)) since at each extreme, then min and max will tend to skew the line). To put it simply: someone really stupid will STILL not pretend that he's "negatively stupid".
The argument "I can make a simple model without using actual humans which shows some kind of bias that vaguely ressembles the result of a paper" doesn't mean that the actual paper is wrong...
Because on the surface, it doesn't make any sense for two sets of "random" numbers between 0-100 selected in pairs to deviate from each other based on whether the first number in the pair was low or not. You would not expect the first number chosen in a pair to influence the second number. Whether the first number was between 0-25 or 76-100, you would expect the second number to be about 50.
So this is obviously some sort of structured randomness that may be entirely justifiable, but the only way to find that out would be to read the two articles that this article purports to summarize for the layman. Instead there's over 1300 words of slop before this sentence, then nearly 700 words of slop after this sentence. Turns out we don't need AI for this. Speaking of random, I don't think that 2000 words is random.
-----
edit:
maybe the point of the papers is that low scorers can't underestimate their abilities - as in they literally don't have enough room? If so, that just means that the Dunning-Kruger affect is unavoidable. But the fact is that people are not choosing numbers at random, they are choosing them based on their expectations. People who got zero questions right and expected 100% are as likely as anyone else from a random number generator, and non-existent from actual people.
edit2:
OK, I've worked it out. I was making the mistake of thinking that they were evaluating absolute performance rather than relative performance. So each of the first numbers in the pair is unique. But that still leaves the fact that the random draw still predictably sits at 50% where the Dunning-Kruger data is around 65% based on the graph. Seems like norming that with the random data would give you better information.
edit3:
> In Dr. Nuhfer’s own papers [...] his team [...] showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
How is "narrower range" not an indication of more accurate self-evaluation? With that, and since people on the higher end of the scale have less room to overestimate their standing, and people on the lower end of the scale have less room to underestimate their standing, wouldn't you expect "Dunning-Kruger"? People on the low end of the scale would have wild swings that would be gated at zero, and people on the high end of the scale would have small swings that would be gated at 100. That would lead to small underestimates at the top, and large overestimates at the bottom. More accurate self-evaluation at the top of the scale is exactly what Dunning-Kruger is about, and the direction of the mistakes is predictable if this is true.
final, tldr:
Honestly, the entire debate is garbled. People are not being asked about their performance on a test, they're being asked about their standing within a sampling of people chosen by the experimenter, something which they have no reason to know anything about other than on the experimenter's word.
I think how people interpret Dunning-Kruger, and the only interesting thing about it, is that people who have more knowledge of a subject are more accurate in their assessment of how much they know about that subject. This seems likely (but not evidently) to be true, due to the range of (relative) self-assessment error being narrower in the top quartile as compared to the bottom quartile. This is what people found intuitive and compelling.
If it is true, the top quartile would tend to small underestimation (because of the narrower range and that they can't choose numbers higher than 100) and the bottom quartile would tend to larger overestimation (because of the wider range and that they can't choose numbers lower than 1.) That the average direction of over- and underestimation is forced by the nature of the evaluation doesn't make the effect any less true.
Mentioning the Dunning-Krueger effect incorrectly is wonderfully meta...
- people not skilled in a thing are bad at estimating their skills
- people are generally bad at estimating their skills
- people skilled in a particular areas often feel they are intellectually fit in other areas
Out of these three I feel like there's some truth in it, at least anecdotally.
[ Unconscious Incompetence ] --> "You don't know what you don't know"
↓
[ Conscious Incompetence ] --> "You know what you don't know"
↓
[ Conscious Competence ] --> "You know, but you have to think about it"
↓
[ Unconscious Competence ] --> "You know it so well, it's second nature"I also noticed people really successful at one thing tend to underestimate other domains, either thinking they are easy or that they can have meaningful input while being essentially novice (see any time Musk opens his mouth for example)
This makes some sense. If people are asked to guess a number between 1 and 6 and then roll a die, the people who roll low are more likely to overestimate and the people who roll high are more likely to underestimate. But the key is precisely how well random data mimics the effect.
This effect would disappear almost completely if they instead of estimating their rank they estimated their score, since then unless the test is so easy the best scores perfectly there will be a lot of room for everyone to overestimate and underestimate themselves.
But as is when the top 10% all estimate themselves to be in the top 10%, you will say they are underestimate themselves since on average the top 10% are in the top 5%. At the same time if the bottom 10% say they are in the bottom 10%, you will say they overestimate themselves since actually on average they are bottom 5%. But both these groups were making the same mistake, and its impossible for that not to happen unless everyone is perfect.
In the article, in the section "The effect is in the noise", just before the graphs, it says they estimated their score. Where did you get that they estimated their rank?
Add negative relationship and negative bias, and the random data will act as intended - hovering randomly around 50%.
But it's not because "some statistical model exhibit a bias that's similar to the result" that this implies "therefore the result is a statistical error"... that's a backward reasonning
I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.
It seems like the overlap between "real psychological effect" and "subtle enough that it requires research to discover" is vanishingly small. I guess that's not really surprising.
I'm at the point honestly, where I don't even consider psychology to be a science anymore.
Besides, deciding what is and isn’t science is a question for the philosophy of science, not science itself.
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
The fact that given X and Y random and independent, that Y-X is correlated with X doesn't disprove the Dunning Kruger. It in fact proves that Y = 1 X is a poor predictor, and the true model is Y = 0 X. In other words, perceived ability (of the human) cannot predict the actual test scores. Which is exactly what DK claims, but to a very extreme effect.
Note that, if there is actual signal (plus noise), e.g., if Y = X + eps; so the actual score is exactly the perceived score plus some added variation, the (Y-X)~X will be uncorrelated. In such case, there will be no DK effect, because the users are good at predicting their actual test scores, plus some constant variation.
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
This is my understanding as well, and as I recall the results of the research were also more nuanced than most people seem to indicate. So not only were the novices with a wider range of self-estimates, the average result of those perceived scores was still below the experts. So it was never that low performers thought they were experts (although this may show up in the raw results with some portion of the population tests), just that on average they perceived their performance to be better than it actually was. And the high competence group while overestimating their results, still thought they'd do better than the low performers.
UPD: probably this one https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...
The article in the post is older though
I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.
Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?
The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.
Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.
...when very mediocre vibe coders think they are brilliant developers....
Do you have a source for this?
In the old plot, the bottom quartile has about a 50 percentage point margin between actual and perceived performance while the new one is 30 percentage points, which is a 50% difference between the old and new curve. The second quartile has 3x more margin in the old version relative to the new one.
Source code is here: https://github.com/pem725/Dunning-Kruger (found here: https://pem725.github.io)
Now it's much more clear. The simulated data tries generating the true relationship between actual and perceived scores from 0.0 to 1.0, and bias in self-reporting from 0% to 100%.
So the output graph should be the average of all these data generating processes, yielding perceived relationship around 0.5 and bias around 50%, with some high variation.
If you have the access, run their Shiny code with these values, and you will see the published plot.
I'd argue that this demonstration is much weaker than "making original no more meaningful than random". It's more that the "simulated 50% bias and 0.5 true correlation looks similar to what DK published", which is also far fetched given the data generation they did.
Note: true random (what they were going for) would cover negative relationships, yielding the random true relationship around 0; and if they wouldn't correct the sign of Bias, it would also average at around 0; yielding a realistic "random" with the slope hovering about 50% for any percentile.
The article is baseless and fluffy
There is so much simple tests that they draw extreme conclusions from in the very first lectures you hear, and when they teach those things as true when its highly debatable its hard to not throw the entire field in the trash.
Of course there is probably some good psychology work done, but you shouldn't have so many shoddy examples in the first course then since that gives people a bad taste for the entire field.
One doesn't need to continue learning a topic to know it's bs when it's bs right from the start.
Sorry, most psychological research is sketchy at best, if not blatantly fraudulent
Physics on the other hand is hard as in unyielding. It is easy to figure out boundaries of physics and map out what is and isn't true, and the few cases were we made a mistake everyone can agree a mistake was made and that formulas needs to be updated since physics is so extremely hard that even a tiny error will get noticed.
Hope that clears it up, physics isn't hard as in difficult, its hard as in rigid. And psychology is soft, not easy.
So, your statement doesn't make sense at all in this discussion, they just said psychology has soft traits, and then you say "ok, so its hard since its soft!". No, soft is difficult, not hard.
So for example, physics is like describing the shape of a metal spoon, and psychology is like describing the shape of a pillow. You can see how describing the shape of the pillow is massively more difficult, because its not fixed, so you have to come up with a language to describe all the ways it can deform and how that would work.
Software engineers tend to repeat the mantra "you cannot make accurate time estimates". That is true regardless of experience level, and everyone seems to be off by about the same amount. So there we have evidence that people overestimate their skills at every level, and its not that different.
https://rationalwiki.org/wiki/Ultracrepidarianism
(the Wikipedia article has that as a subheading on the Ne supra crepidam page instead: https://en.wikipedia.org/wiki/Ne_supra_crepidam#Ultracrepida...)
But I'd say it's a subtype of the Dunning-Krueger effect rather than part of the general definition.
Personally I've always thought of Dunning-Krueger as an extension of the first definition here. Newbies/amateurs overestimating their skills and experts underestimating them, though I feel like the second definition is probably more true of human behaviour.
"I seem, then, in just this little thing to be wiser than this man at any rate, that what I do not know I do not think I know either."
I can’t remember of DK suggests some sort of effect where the expert has undue self-doubt, though…
There is a brief bit where they attribute that observation to other causes:
In making these predictions, we felt that we could account for an anomaly that appeared in all three previous studies: Despite the fact that top-quartile participants were far more calibrated than were their less skilled counterparts, they tended to underestimate their performance relative to their peers. We felt that this miscalibration had a different source then the miscalibration evidenced by bottom-quartile participants. That is, top-quartile participants did not underestimate themselves because they were wrong about their own performances, but rather because they were wrong about the performances of their peers. In essence, we believe they fell prey to the false-consensus effect (Ross, Greene, & House, 1977). In the absence of data to the contrary, they mistakenly assumed that their peers would tend provide the same (correct) answers as they themselves—an impression that could be immediately corrected by showing them the performances of their peers. By examining the extent to which competent individuals revised their ability estimates after grading the tests of their less competent peers, we could put this false-consensus interpretation to a test.
Link if you need it:
https://www.researchgate.net/publication/12688660_Unskilled_...
This I think is a separate phenomenon, maybe Nobel Disease but there might be a more general term, for example that includes celebrities.
For example, there are always at least a few people who you have to consult regularly because their tickets are always written how they perceive a problem should be solved rather than explaining the underlying problem to be solved. Then there are the sales and marketing types who like to dictate how the UX of the front-end should be because they like to retell anecdotes to prospective clients about how they solved a problem. Then there's the guy who hands you a spreadsheet he's maintained for years and tells you he wants you to put something exactly like it in the flagship product as a new feature.
And of course there's a relevant xkcd: https://xkcd.com/2501/
See the Nostalgia Critic review of the Wall. Or the Alex Meyers review of Obsession.