In reality, they cheat when a culture of cheating makes it no longer humiliating to admit you do it, and when the punishments are so lax that it becomes a risk assessment rather than an ethical judgment. Same reason companies decide to break the law when the expected cost of any law enforcement is low enough to be worth it. When I was in college, overt cheating would be expulsion with 2 (and sometimes even 1 if it was bad enough) offenses. Absolutely not worth even giving the impression of any misconduct. Now there are colleges that let student tribunals decide how to punish their classmates who cheat (with the absolutely predictable outcome)

The two solutions to this are (1) as some commenters here are suggesting, give up entirely and focus only on quality of output, or (2) teach students to care about being more than appearance. Make students want to write essays. It is for their personal edification and intellectual flourishing. The benefits of this far surpass output.

Obviously this is an enormously difficult task, but let us not suppose it an unworthy one.

I suppose there are other fields where the degree might be used mostly as a filtering mechanism, where cheating through graduation might get you a job doing work different than your classes anyway. However, even in those cases it's hard to break the habit of cheating your way around every difficult problem that comes your way.

The real problem is students and universities have collectively bought into a "customer mindset". When they do poorly, it's always the school's fault. They're "paying customers" after-all, they're (in their mind) entitled to the degree as if it is a seamless transaction. Getting in was the hardest part for most students, so now they believe they have already proven themselves and should as a matter of routine after 3-4 years be handed their degree because they exchanged some funds. Most students would gladly accept no grades if it was possible.

Unfortunately, rather than having spines, most schools have also adopted a "the customer is always right" approach, and endlessly chase graduation numbers as a goal in and of itself and are terrified of "bad reviews."

There has been lots of handwringing around AI and cheating and what solutions are possible. Mine is actually relatively simple. University and college should get really hard again (I'm aware it was a finishing school a century ago, but the grade inflation compared to just 50 years ago is insane). Across all disciplines. Students aren't "paying for a degree", they're paying to prove that they can learn, and the only way to really prove that is to make it hard as hell and to make them care about learning in order to get to the degree - to earn it. Otherwise, as we've seen, the value of the degree becomes suspect leading to the university to become suspect as a whole.

Schools are terrified of this, but they have to start failing students and committing to it.

Students had and still have the option to collectively choose not to use AI to cheat. We can go back to written work at any time. And yet they continue to use it. Curious.

[...]

> Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression.

Yeah, not sure the conclusion of the article really matches the data.

Students were invited to talk to an AI. They did so, and having done so they expressed a clear preference for written exams - which can be taken under exam conditions to prevent cheating, something universities have hundreds of years of experience doing.

I know some universities started using the square wheel of online assessment during covid and I can see how this octagonal wheel seems good if you've only ever seen a square wheel. But they'd be even better off with a circular wheel, which really doesn't need re-inventing.

I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof.

Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes.

I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. I had three hours to do the exam. Full stop. If there was a whiff of cheating, you were expelled. Do not pass go. Do not collect $200.

Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm.

You were expected to learn the gd material. The university thanks you for your donation.

I feel like i'm taking crazy pills when I read things about trying to "adapt" to AI. We already had the solution.

Don't tell me about GenZ. I had oral exams in calculus as undergrad, and our professor was intimidating. I barely passed each time when I got him as examiner, though I did reasonably well when dealing with his assistant. I could normally keep my emotions in check, but not with my professor. Though, maybe in that case the trigger was not just the tone of professor, but the sheer difference in the tone he used normally (very friendly) and at the exam time. It was absolutely unexpected at my first exam, and the repeated exposure to it didn't help. I'd say it was becoming worse with each time. Today I'd overcome such issues easily, I know some techniques today, but I didn't when I was green.

OTOH I wonder, if an AI could have such an effect on me. I can't treat AI as a human being, even if I wanted to, it is just a shitty program. I can curse a compiler refusing to accept a perfectly valid borrow of a value, so I can curse an AI making my life difficult. Mostly I have another emotional issue with AI: I tend to become impatient and even angry at AI for every small mistake it does, but this one I could overcome easily.

In my graduate studies in Germany, most of my courses used oral exams. It's fine, and it's battle-tested.

When I was doing a lot of hiring we offered the option (don’t roast me, it was an alternative they could choose if they wanted) of a take-home problem they could do on their own. It was reasonably short, like the kind of problem an experienced developer could do in 10-15 minutes and then add some polish, documentation, and submit it in under an hour.

Even though I told candidates that we’d discuss their submission as part of the next step, we would still get candidates submitting solutions that seemed entirely foreign to them a day later. This was on the cusp of LLMs being useful, so I think a lot of solutions were coming from people’s friends or copied from something on the internet without much thought.

Now that LLMs are both useful and well known, the temptation to cheat with them is huge. For various reasons I think students and applicants see using LLMs as not-cheating in the same situations where they wouldn’t feel comfortable copying answers from a friend. The idea is that the LLM is an available tool and therefore they should be able to use it. The obvious problem with that argument is that we’re not testing students or applicants on their abilities to use an LLM, we’re using synthetic problems to explore their own skills and communication.

Even some of the hiring managers I know who went all in on allowing LLMs during interviews are changing course now. The LLM-assisted interviewed were just turning into an exercise of how familiar the candidate was with the LLM being used.

I don’t really agree with some of the techniques they’re using in this article, but the problem they’re facing is very real.

Where do we go from there? At some point soon I think this is going to have to come firmly back to real people.

As an aside, I'm surprised oral exams aren't possible at 36 students. I feel like I've taken plenty of courses with more participants and oral exams. But the break even point is probably very different from country to country.

If this is the only way to keep the existing approach working, it feels like the only real solution for education is something radically different, perhaps without assessment at all

No, we do not want to eliminate the pen and paper exam. It works well. We use it.

The oral exam is yet another tool. Not a solution for everything.

In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actually understood their own project and did not rely solely on an LLM. (We do allow them to use LLMs, it would be stupid not to.)

The students who did badly in the oral exam were exactly the students who we expected to do badly in the exam, even though they aced their (team) project presentations.

Could we do it in person? Sure, we could schedule personalized interviews for all the 36 students. With two instructors, it would have taken us a couple of days to go through. Not a huge deal. At 100 students and one instructor, we would have a problem doing that.

But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.

Reminder: This professor's school costs $90k a year, with over $200k total cost to get an MBA. If that tuition isn't going down because the professor cut corners to do an oral exam of ~35 students for literally less than a dollar each, then this is nothing more than a professor valuing getting to slack off higher than they value your education.

>And here is the delicious part: you can give the whole setup to the students and let them prepare for the exam by practicing it multiple times. Unlike traditional exams, where leaked questions are a disaster, here the questions are generated fresh each time. The more you practice, the better you get. That is... actually how learning is supposed to work.

No, students are supposed to learn the material and have an exam that fairly evaluates this. Anyone who has spent time on those old terrible online physics coursework sites like Mastering Physics understands that grinding away practicing exams doesn't improve your understanding of the material; it just improves your ability to pass the arbitrary evaluation criteria. It's the same with practicing leetcode before interviews. Doing yet another dynamic programming practice problem doesn't really make you a better SWE.

Minmaxing grades and other external rewards is how we got to the place we're at now. Please stop enshittifying education further.

Also, with all the progress in video gen, what does recording the webcam really do?

Ask about any teacher, scalability is a serious issue. Students being in classes above and under their level is a serious issue. non-interactive learning, leading to rote memorization, as a result of having to choose scaling methods of learning is a serious issue. All these can be adjusted to a personal level through AI, it's trivial to do so, even.

I'm definitely not sold on the idea of oral exams through AI though. I don't even see the point, exams themselves are specifically an analysis of knowledge at one point in time. Far from ideal, we just never got anything better, how else can you measure a student's worth?

Well, now you could just run all of that student's activity in class through that AI. In the real world you don't know if someone is competent because you run an exam, you know if he is competent because he consistently shows competency. Exams are a proxy for that, you can't have a teacher looking at a student 24/7 to see they know their stuff, except now you can gather the data and parse it, what do I care if a student performs 10 exercises poorly in a specific day at a specific time if they have shown they can do perfectly well, as can be ascertained by their performance the past week?

This sounds as though it was written by an LLM too.

On the other hand, can an AI exam really simulate the conditions necessary for improving at this skill? I think this is unlikely. The students' responses indicate not a general lack of expertise in oral communication but also a discomfort with this particular environment. While the author is making steps to improve the environment, I think it is fundamentally too different from actual human-to-human discussion to test a student's ability in oral communication. Even if a student could learn to succeed in this environment, it won't produce much improvement in their real world ability.

But maybe that's not the goal, and it's simply to test understanding. Well, as other commenters have stated, this seems trivially cheatable. So it neither succeeds at improving one's ability in oral communication nor at testing understanding. Other solutions have to be thought of.

LLM oral exams can provide assessment in a student's native language. This can be very important in some scenarios!

Unlimited attempts won't work in the presented model. No matter how many cases you have, all will eventually find their way to the various cheating sites.

There is no silver bullet. There's no solution that works for all schools. Strategies that work well for M.I.T. with competitive enrollment and large budgets won't work for a small community college in an agricultural state, with large teaching loads per professor, no TAs, and about 15-25 hours of committee or other non-teaching work. That was my situation.

Teaching five courses and eight sections, 20-30 students per section, 10-20 office hours every week (and often more if the professor cared about the students), leaves little time for grading. In desperation I turned to weekly homework assignments, 4-6 programming projects, and multiple choice exams (containing code and questions about it). Not ideal by any means, just the best I could do.

So I smile now (I'm retired) when I hear about professors with several TAs each, explaining how they do assessment of 36 students at a school with competitive enrollment.

I could accept this for a 300 students class, but 36? When I got my degree, ALL exams had an oral component, usually more than 30 minutes long. The prof and one or two TAs would take a couple days and just do it. For 36 students it’s more than doable. If I was a student being examined by an LLM I would feel like the professor didn’t care enough to do the work.

This is to be expected. The big commercial LLMs generally respond with text that agrees with the user.

> But here's what's interesting: the disagreement wasn't random. Problem Framing and Metrics had 100% agreement within 1 point. Experimentation? Only 57%.

> Why? When students give clear, specific answers, graders agree. When students give vague hand-wavy answers, graders (human or AI) disagree on how much partial credit to give. The low agreement on experimentation reflects genuine ambiguity in student responses, not grader noise.

The disagreement between the LLMs is interesting. I would hesitate to conclude that "low agreement on experimentation reflects genuine ambiguity in student responses." It could be that it reflects genuine ambiguity on the part of the graders/LLMs as to how a response should be graded.

Part 2 is that when you are ready, an examiner sits with you, looks over your stuff and asks questions about it, like clarifications, errors to see if you can fix them, fake errors to see if you can defend your solution, sometimes even variations or unrelated questions if they are on the fence as to the grade. Typically that takes 3-10 minutes per person.

Works great to catch cheating between students, textbook copying and such.

Given that people finish asynchronously you don't need that many examiners.

As to being more stressful for students I never understood this argument. So is real life.. being free from challenge based stress is for kindergarteners

In a presentation, you are in control. You decide how you will present the information and what is relevant to the theme. Even if you get questions, they will be related to the matter at hand that you need to dominate in order to present.

In oral exams, the pressure is just too great. I doubt it translates to a proper job. When I'm doing my job, I don't need to come up with answers right there on the spot. If I don't remember something, I have time to think it through, or to go and check it out. I think most jobs are like this.

I don't mind the pressure when something goes wrong in the job and needs a quick fix. But being right there, in an oral exam, in front of an antagonistic judge (even if they have good intentions) is not really the way to show knowledge, I think.

(I invented some kind of metric based on a centered gaussian around a country ahaha)

One big issue that I had is that the system asked for a number in dollars, but if I answer $2000,2000,2000 per agent per month, the answer was always the same, I cannot accept a number, give it in words, after many tries I stopped playing, it wasn't clear what it wanted.

I could see myself using the system. With another voice as it was kind of agressive. More guidelines would be needed to know exactly how to pass a question or specify numbers.

I don't know my grade, so I don't know how much we can bullshit the system and pass

And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thanks you for your donation" slogan doesn't sound too hot all of a sudden.

It's like some malicious compliance take on both teaching and studying. Which shouldn't even be surprising, considering the circumstances of the professors e.g. where I studied, as well as the students'.

Mind you, I was (for some classes) tested the same way. People still cheated, and grading stringency varied. People still also forgot everything shortly after wrapping up their finals on the given subjects and moved on. People also memorized questions and compiled a solutions book, and then handed them down to next year's class. Because this method does jack against that on its own. You still need to keep crafting novel questions, vary them more than just by swapping key values, etc.

> I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading.

Do you, like, really think this is the best way to assess someone's ability? Can't we find a place between the two extremes?

Personally, I'd go with a school-provided computer with a development environment and access to documentation. No LLMs, except maybe (but probably not) for very high-level courses.

For comparison we had lengthy sessions in a jailed terminal, week after week, writing C programs covering specific algorithms, compiling and debugging them within these sessions and assistants would follow our progress and check we're getting it. Those not finishing in time get additional sessions.

Last exam was extremely simple and had very little weight in the overall evaluation.

That might not scale as much, but that's definitely what I'd long for, not the Chuck Norris style cram school exam you are drawing us.

Also, IMO oral examinations are quite powerful for detecting who is prepared and who isn't. On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those.

You have a very weird idea of education if a teaching method that results in a 99% failure rate is seen as good by yourself. Do you imagine a professional turning out work that was 99% suboptimal?

The old ways do not scale well once you pass a certain number of students.

> And here is the delicious part: you can give the whole setup to the students and let them prepare for the exam by practicing it multiple times. Unlike traditional exams, where leaked questions are a disaster, here the questions are generated fresh each time. The more you practice, the better you get. That is... actually how learning is supposed to work.

They're even more possible if you do an oral exam only on the highest grades. That's the purpose, isn't it? To see if a good, very good, or excellent student actually knows what they're talking about. You can't spare 10 minutes to talk to each student scoring over 80% or something? Please

It depends on how frequent and how in-depth you want the exams to be. How much knowledge can you test in an oral exam that would be similar to a two-hour written exam? (Especially when I remember my own experience where I would have to sketch ideas for 3/4th of the time alloted before spending the last 1/4th writing frenetically the answer I found _in extremis_).

If I were a teacher, my experience would be to sample the students. Maybe bias the sample towards students who give wrong answers, but then it could start either a good feedback loop ("I'll study because I don't want to be interrogated again in front of the class") or a bad feedback loop ("I am being picked on, it is getting worse than I can improve, I hate this and I give up")

For the use of LLM in classes: I understand the reasoning, but I found LLMs to be extremely educational for parsing through dense material (eg parsing an NTSB report for an Uber self-driving crash). Prohibiting students from using LLMs would be counterproductive.

But I still want students to use LLMs responsibly, hence the oral exam.

CFR 46.104 (Exempt Research):

46.104.d.1 "Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods."

https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-...

So while this may have been a dick move by the instructors, it was probably legal.

Even for distance education though, proctored testing centers have been around longer than the internet.