Things I learned about how people use AI after 1800 people

6 points by Ozzie-D 11 days ago | 2 comments

I'm running a an AI fluency assessment tool and after 1800 real users who interact with our chat bot 20-40 minutes; here are some weird things we found out.

1. The least AI-fluent professionals overestimated their score by 40 points. The most fluent underestimated by 27. 67 point Dunning-Kruger gap.

2. Product managers outscore engineers on AI fluency (59.2 vs 53.7). Applied judgement beats technical knowledge.

3. HR people (ironically who make hiring decisions using AI) understand AI the least, with the lowest AI fluency score.

4. People consistently say they are good with AI, but 2 out 3 fail to reach even proficient level.

5. The average AI fluency score across 1,800 professionals is 48 — squarely in the Developing tier. Most people use AI regularly but without systematic practice.

6. Company-wide AI training assumes everyone starts from the same place. The data says starting points vary by 5x within the same team.

7. Same company, same tools, same training budget. The gap between the least and most AI-fluent employee in one team was 82 points (15 to 97).

If you are sceptical (rightly so) you can review our methodology here: https://aisa.to/methodology

josefcub 10 days ago |

You know, I took this when there were only a handful of people on the leaderboard. Things I didn't know:

20-40 minutes is 2 hours if you're like me and want to make sure you're actually thorough answering questions. Aisa (the agent) was great at asking pertinent questions and keeping the conversation somehow both on-topic and on-going. It was not nearly as laborious as I thought it would be, despite how long it took.

I scored 89. I'm still in the top 30 on the leaderboard. I have no idea how I scored that with as little experience at work as I have with AI. At the time AISA first debuted here, we hadn't even gotten access to agentic models at my employer, just chatbots. I learned literally everything at home with local models.

I'm not going to argue a high score, but that feeling of disbelief is still real. I think this is a great product for evaluating your personal skills according to a fairly broad but well-defined rubric. The fact that you get to read the analysis itself and why you were graded as you were is great for self-improvement.

If I had an ask, it would be to actually download that analysis and the original conversation log. It was comprehensive enough that I'd love to turn it into actionable steps for improvement without having to copy-paste so extensively.

Thanks for making an evaluation system that works so well!

Ozzie-D 10 days ago | |

Thank you for sharing your experience. The calibration has "settled" a bit more since then and you might score perhaps a bit lower (or not) but might be worth a retry especially if you have learned new things.

For example we have a fatigue model (that tracks how bored the user is and adjusts) a retrain model (that retrains on the frontier knowledge in AI everyweek) a golden-shadow model (that looks at what you did NOT say to make a prediction about how good you might be on HOW you say) and a learning-agility model. These are all new.

It's not perfect but it's improving with every assessment and I have a ton of product backlog notes to make it even more advanced.