This approach does not really solve the core issue. In practice, students often do poorly when evaluation is concentrated in one end of term exam. It also pushes many students to cram at the end of the term instead of learning steadily.

A better approach is to rethink what we assess and how we assess it. Research shows that the design of assessments plays an important role in academic integrity. Assignments that require original thinking and regular engagement can reduce incentives to cheat and improve learning outcomes.

https://www.sciencedirect.com/science/article/abs/pii/S22119...

I got diagnosed with cancer just before finals in the first semester of my senior year. Sure, it kill my chances at graduating Summa Cum Laude, and I didn’t make the Dean’s list that semester even though I worked my ass off, as usual. Frustrating, but that’s life. I should not, however, have failed that semester, which I would have if only the final week’s assignments were counted. People have bad weeks. In most white collar jobs I’d have probably been able to take some time for myself, maybe given someone else my most urgent tasks, and likely been given plenty of leeway. Even doctors, lawyers, etc. People deserve to have bad weeks without losing months of work.

Let's just focus on CS for a moment because this is HN and I have a CS degree. How do you evaluate a student's ability to implement a piece of non trivial software following a specification that would take, say, over five thousand lines? This is a fairly common type of coursework for CS students. It's not possible to do it in one exam setting. Well, unless you can type continuously without having to stop to think or check your work.

What will likely happen if you have to force all evaluation into one exam is you will get a Leetcode style programming quiz, that tests something, except that something is only barely connected to your daily job after college. Or you end up with something like a multiple choice exam that is again disconnected from your work and can be gamed by studying just for the exam.

This exact line of argument that we should just abolish coursework keeps coming up on HN. Courseworks exist for a purpose. You can't just throw it away and pretend it never did anything useful.

Why not multiple exams? In fact, why not many exams?

Sure, it requires more resources, but it shouldn't require much more:

- We've had multiple exams before AI, and I don't see how AI makes it any harder. Obviously these are closed-book

- Schools should already be banning phones in class (and colleges have insane tuitions, they can afford more exams)

- The students who go out of their way to cheat - as long as they're a minority, let them. Why not? Either they'll fail later in life, or they didn't need to learn the material because they're pathological fakers (even if you won and forced them to learn the material, they'd probably still fake their way out of using it). Then, I doubt you need much proctoring to ensure that most students don't cheat, because most of the smart students are generally smart enough to know that actually learning the material is probably important (or if the material is probably not important, it doesn't matter if the students all cheat...)

Meanwhile, downsides of one exam:

- Disadvantages students who get overly stressed about unrecoverable exams, or have a particularly bad day on the exam

- Many students will blow off the (ungraded) assignments and put off actually learning until the end

- Less graded content (especially if the exam isn't overly long, which would disadvantage some students)

I don't disagree with you that a reasonable way to cope with the current problems is to ensure everything that "counts" is done in a controlled environment, but pedagogy and its goals are vast.

There are things you learn from spending several days structuring a 20-page argument that you will not learn (and cannot assess) from oral examination or a 5-paragraph essay written in a blue book.

Schools stopped doing that because students largely refuse to prepare. Testing throughout the year is like a CI pipeline and is shown to work better for the median student.

Throw in some oratory presentations as well and that sounds like a curriculum.

Students are very grade-motivated and unfortunately they rarely do the homework assignments if they are not worth points.

At-home coding projects, writing essays, etc also exercise different skiils than you can test for in a 2 hour written exam. It's unfortunate that due to rampant AI cheating, we can no longer reward the students who put in the work and develop these skills.

Why a single test at the end of the semester? Why not allow the student to demonstrate mastery at anytime during the semester when they are ready? Then they can move on to the next objective, or, if they fall short, continue to study until they meet the objectives.

Of course, creating good exams is difficult, but you have to do that either way.

Unfortunately this approach is known to be really bad for some students. Continuous assessment is bad for other students.

I was lucky in that my education last millennium was almost entirely make-or-break final exams, which suit me well. I’m bad at routinely completing little assignments, but shine under crazy pressure. Other students are the opposite.

We’ve tried both kinds over time and trend. Neither is perfect.

> one exam at the end of the term where the student has an opportunity to demonstrate mastery of the course's subject matter. The resulting grade depends on that, and that alone.

I love this idea. And if a student is having a really bad day, or their dog just died, or they have bad cramps, or they have a hard time dealing with the intense stress of your entire grade being decided in one exam... well, those loser students can just fuck right off.

The purpose of grades is to punish students, something which they are keenly aware of. Remove grades from the equation and hold students back until they have mastered the material and they will cease cheating.

If someone knows 80% of the topics on an exam like the back of their hand and doesn't know the other 20% they shouldn't get a B, they should pass the subjects they know and be asked to retake and relearn the subjects they don't know.

When people know they can make mistakes and the result is not a perpetual black mark on their record (any grade not an A) but they are given the chance to improve and demonstrate this improvement then perhaps they might be more willing to admit and understand mistakes instead of cheating.

What exactly would the goal of this change be?

Maybe you can call it an interview…

Came here to say the same thing. The AI problem is functionally no different to the paid essay writers. Grade everything at face value, and then have people write essays under exam conditions for grading.

Even before LLMs, there was a _lot_ of deception and cheating in university. I -- and I do not say this with pride -- used to write essays for my classmates for money. In my own defense, I needed the money. I also know that in addition to homework for money, many fraternities and sororities kept copies of prior exams and assignments, and getting access to these was one of the perks of membership. Knowing what kind of questions to expect (let alone the exact questions) can easily give someone a few extra IQ points for free.

Personally, I felt that the drive to automate the parts of the professors' workloads that mattered (i.e. teaching and grading and evaluation and research), only so that they can be given work that matters less the more they do it (i.e. publishing slightly different flavors of the same paper, to meet KPIs), was oddly perverse.

The multiple-choice test and the puzzle-solving test and really any standardized test can be exploited by any group that is sufficiently organized. This is also true in corporate interviewing where corporations think (or pretend) that they are interviewing an individual, whereas they are actually interviewing a _network_ of candidates who share details about the interviewers and the questions. I know people who got rejected in spite of getting all the interview questions correct (the theory is that nobody can do that well, so they must have had help from previously rejected/accepted candidates).

The word "trust" shares a root with the word "tree" and "truth" and "druid". Most exams and interviews are trying to speed-run trust-building (note that "verification" is from the latin word that means "true"). If trust and truth are analogous to "tree", then we are trying to speed-run the growth of a tree -- much like the orange tree, in the film, _The Illusionist_. And like the orange tree, it is a near-complete illusion, a ritual meant to keep the legal department and HR department happy.

The LLMs have simply made the corruption of academia accessible to _all_ students with an internet connection (EDIT: and instantaneous and cheap, unlike a human writer).

There has never been a shortcut to building trust. One cannot LLM their way into being a (metaphorical) druid.

I do not look forward to the Voight-Kampff tests that will come to dominate all aspects of online and asynchronous human interaction.

Note that, short of homework/classwork that _can't_ be gamed by an LLM (for some fundamental reason), even the high-quality honest students will be forced to cheat, so as to not be eclipsed by the actual low-quality cheating students[0].

I imagine that we may end up wrapping around to live in-person dialectics, as were standard in the time of Socrates and Parmenides[1]. If so, this should be fun.

[0]: If left unaddressed, we may see a bimodal distribution of great and terrible students graduating college, with those in between dropping out. If college is an attempt to categorize and rank a population, this would be a major fault in that mechanism.

[1]: Not to the exclusion of the other kinds of tests, writing is still important, critical even. But as a kind of verification-step, that should inform how much the academic community should trust the writing (I can imagine that all the writers here are experiencing stage-fright as they are reading these words).

I had fun with those because they only care about the quality of the writing not the content so I would make sure that none of my facts or references were real.

Being able to write in a formulaic manner is a skill though. Not being able to write properly is not great for communication.

Right? It’s the obvious question:

why are they using software to detect software?

I can detect AI written prose in less than five seconds; I would expect a trained teacher to be able to do that as well.

I think peak writing was probably greatest generation. We lost the art of correspondence in the years since.

This is true, I know someone that has read multiple versions of the bible and their writing style became very similar to that. There's a term for it, I just forgot what the term was

em-dashes have been house style for where I've worked for over a couple decades. If people don't like it, F them. I'm not going to change how I write because people may think it make me more AI-like.

Here's one vote for just be the witch if that's what people need from you.

Just make it be what you want to say and how you want to say it. And when they come after you, shame them to the best of your ability or treat them like they are not there.

Article is about student writing essay by herself, but using ai detector to prevent false accusation.

Do you have any evidence to back this up or is it speculative?

My institution subscribes to TurnItIn's AI detector. The documentation is quite clear that the system is tuned in a manner that produces a significant number of false negatives and minimizes false positives. They also state that they don't report anything under "20% AI-generated" content.

So the marketing I've seen is intended to reassure skittish administrators that the software is not going to generate false accusations.

That being said, I have no idea whether the marketing claims are true. The software is a black box.

Avoid the theory-heavy disciplines. You won't be told what to think (as often) if you take History and Geography rather than Sociology and Gender Studies.

An essay is a good gauge of how one can organise their thoughts, argue a position, respond to a stimulus.

In short it’s a good way measure thinking.

Public Schools. I think terror there is built as a feature, not a bug. So be afraid.

But keep in mind, it may have always been this way. God bless those few cool teachers in each school who are aware of this and work to rescue a few who need it.

Love changes everything. Good teachers matter.

> essay writing is no longer a viable form of assessment.

Of course it is. In person, with an unseen prompt/question. By hand or not doesn’t really matter as we can airgap or just monitor via software when in class.

> This is what terrifies me about the public school system.

This has nothing to do with Public School in particular. This is impacting private and university education too.

>> But if a vendor makes a pretty believable claim that there are repetitive statistical patterns in LLM output, it's all of sudden treated the same as palm reading.

That's what fortunetellers do. The problem isn't guessing correctly about AI content in writing. The problem is false positives. That's what puts it in the same category is predictive policing scam software. And fortunetelling.

Eliminating any statistically significant difference between a high-quality human-written text and LLM-written text is exactly what the LLMs are being trained for. At this point, "text is low quality, therefore must be human" is a much stronger signal.

You can detect if texts from a year ago used AI based on statistical patterns. Nobody is taking issue with that. But once you tell people "we will run these tests to detect if your future submissions are using AI" you create an adversarial environment and your statistical methods will continuously break. Not because statistics is broken, but because you are trying to hit a moving target that doesn't want to be hit.

That's not like detecting thoughts via fMRI, it's like detecting tomorrows malware with yesterday's malware signatures. Or like researchers making a vaccine against the common cold

And the obvious proposal to fix that has been made multiple times in this thread: don't make take-at-home tasks part of the grade. Instead of trying to punish what you can't reliably detect, take away the incentive to do it in the first place

> When we have a press release from a university about how researchers can detect thoughts via fMRI, we have no issue with the claim.

Different people. I for one have always claimed that fMRI is too coarse-grained for detailed thought detection.

If AI detection "sometimes fails", it doesn't "work". It works well enough to convict someone with other evidence, but when there's no other evidence nor an attempt to get any, it has no good use.

What I propose is simple: grade only closed-book exams, and hold students' phones during the exams. Students don't need 1:1 monitoring, it's the same as 10-20 years ago.

> that the cost of catching cheaters is significantly higher than the cost of cheating

This is tackling the problem from the wrong direction. The right direction would be to make it harder to cheat in the first place. For example: if the student submits an essay, and that student is able to coherently and accurately answer any questions asked about the essay in a face-to-face conversation, then that student is probably the genuine author of that essay.

Every kind of examination should be proctored.

> using a calculator

Students who are at a level where they'd be learning to do the computations a calculator does, shouldn't have graded homework. And even at that level, real mathematics is more than just computation.

> getting answers from previous tests

Decades ago, my teachers and professors knew advanced tricks for this, like "not just reusing the test questions from last year". Sometimes they even changed the constants in math questions between sections of the class.

Reading previous tests (including correct answers) was never considered cheating, or even slightly unethical, in my education. In fact, one of our professors had this party trick of working through all the answers for a past-year exam (perhaps multiple of them; I can't recall the details, but certainly much faster than students were expected to work things out under exam conditions) in the space of a single lecture, near the end of the course. Students were meant to see this and learn from it (as well as be impressed).

> resubmitting assignments

Why would you ever not notice this?

We can't, and neither can the machines that people build and/or use for "detection." Everyone in this thread also needs to recognize the entrenched differences between secondary educators, who have wholeheartedly adopted AI products into their teaching workflow, and tertiary educators, who have adopted them only by necessity. "By necessity" in this case means "having to spend a ton of time dealing with, talking about, and learning about this nonsense."

The discourse around "cheating" with these products has always been a mistake. We should have characterized them less as "cheating machines" and more as "expediency machines." Because once you're invested in describing students as having academic dishonesty issues rather than skill issues, you've made it an administrative problem. You never come back from that.

For mine, we lost the issue long ago when accountability culture won. We should never have bothered with the idea that "mechanics, grammar, and proofreading" should be part of a "rubric" that "assessed outcomes" for "good writing." We should have just said "we don't care if you don't think this is worthwhile, because your time is worth nothing." The last two years of student labor certainly suggests this.

Sure. And my point is that the assignment is poorly conceived if an LLM's output can appear to "have ideas" that satisfy the prompt. Last I checked, they don't do a good job of modeling a specific, non-notable person within particular constraints, and then all the relevant life experiences of that person. An LLM essay should be human-detectable for the same reasons that one from an essay mill would be.

Is that not pointless now? The point of writing was previously to communicate our thoughts and ideas to other people. Now and going forward that is unnecessary. The most efficient and effective way for us to communicate our thoughts and ideas is to have an agent organize and write them down for us.

I totally agree with this. I would add that it's well beyond the discussion boards. It's probably most clear there and it's well possible we learned it there and then took it into our social interactions everywhere, but the majority of my irl interactions—except with my closest friends—are sorta like this. Sometimes I think its ADHD, other times I think it could be any number of things, but I think to say anything that isn't dead simple (or in dead agreement with the other person), you need a few sentences. Often, you need to hear the third sentence before the first will make sense. But if you get distracted by the first one or can't suspend your disagreement enough to get to the third you will think the person is mistaken. You'll think that about both their first point and the larger one, which you didn't really hear or even get to but thought you did. So the speaker does the hedging each sentence in hopes of getting to the third (or whatever) sentence.

>too much hedging and over-specifying to try to head off shitposting by bad or bad-faith readers.

yeah but if the OP doesn't do that and you confront their argument they can retreat into definitions and ambiguity without addressing your rebuttal. i think its good manners to be hyper-specific particularly on HN where there tend to be a lot of martian brained people who need it to engage with you. the fuzziness just won't do.

This is all communication no?

If people do not share the same context, then they will come up with different interpretations of the same content.

In communities with more homogenous understanding of the context, people are able to get into the details more effectively.

Those communities tend to also be impervious to outsiders, or newbies, because the use of terms/jargon that speeds up conversation abound.

Despite being a different kind of writing, there are some interesting parallels with the article in what you wrote here

I just wanted to tell you that I read your comment immediately after writing mine and it's almost eerie how similar they are. There's the proof, if we needed any!

There's a tension (imo) between deciding to only spend time trying to talk to people who immediately agree with you or are open to hearing you out vs those who immediately disagree such that they will fight hard to not hear, not understand, misinterpret, or "not have time for this". The latter is a specific form of disagreement where they've "noise-canceled" the possibility of learning or understanding (even if it would be perfectly reasonable for them to disagree with it afterward).

Is your life easier to not waste time on them? I guess. But obviously you're going to put yourself in a similar bubble, and to whatever extent the issue is important it's now become undiscussed. As you've hinted at, they could be right and you wrong, but the difference is (at least in the premise) that one is willing to talk and listen and so really only one side has the potential to change and it's not based on the merit of the argument—because of course no conversation took place. How hard does one try to encourage someone else to listen? Or rather continuing pursuing a conversation that's being denied? That's the tension. I don't know other than it seems like the side unwilling to listen wins a little bit each time they've successfully evaded it and wins a little more when the other has decided to let it go. I don't just mean they've won a proverbial argument, I mean the issue or decision in question tilts toward their side.