AI Detection in Education: What Schools Are Getting Wrong

• →Schools are using AI detectors as pass/fail verdict tools. They're probabilistic instruments that were explicitly never designed for that.
• →Non-native English speakers get flagged disproportionately — not as an edge case, but as a documented, structural bias in perplexity-based tools.
• →Single-number scores are the wrong format for academic integrity decisions. Category breakdowns surface the right questions instead of just triggering penalties.
• →The most reliable AI signal in student work isn't a score. It's the absence of struggle. Real student writing has friction. AI output is too smooth.
• →Detection should be the start of a conversation, not a substitute for one.

I want to be careful here, because I'm going to be critical of how AI detection is being used in schools — and I don't want that to read as a defense of academic dishonesty. Students submitting AI-generated work as their own is a real problem. Worth taking seriously. The question I'm asking is whether what's currently being done about it is actually working — or whether it's producing a different set of problems while mostly failing at the original one.

My read: it's making things worse in some important ways. Not because detection is useless, but because the tools are being deployed with a confidence level they were never built to support. And the students paying the price aren't always the ones cheating.

The false positive problem isn't theoretical — it's documented

In 2023, Turnitin launched its AI detector with significant fanfare. Within months, a Washington Post investigation found it was incorrectly flagging human-written essays at rates that would translate to thousands of false accusations in large school districts. Turnitin added a disclaimer — the tool shouldn't be used as sole evidence in academic integrity proceedings. Most instructors using it either didn't see that disclaimer or didn't change their practice.

The students who get flagged disproportionately aren't random. A 2023 study by Liang et al. at Stanford found that AI detectors misclassified non-native English writing as AI-generated at significantly higher rates than native writing — a direct result of how perplexity scoring works. Non-native speakers writing carefully to avoid errors produce text with simpler, more predictable vocabulary and sentence structures. That statistical profile overlaps with how some language models write. The detector can't tell the difference, so it flags both.

That's not a minor calibration issue. It's a structural problem with using these tools as adjudicators rather than investigation triggers. A tool designed to catch cheating that disproportionately penalizes students already at a disadvantage isn't doing the job it was deployed for.

The verdict problem

Here's what I think is the core mistake: schools are treating AI detector scores as findings of fact. "93% AI-generated" becomes an accusation in an academic integrity hearing. The student has to defend themselves against a probabilistic number produced by a tool that — by its own documentation — was never meant to function that way.

Probabilistic instruments aren't verdicts. A weather forecast saying 93% chance of rain is not a guarantee it'll rain. A detector returning 93% AI confidence is not saying "this was AI-generated" — it's saying "this text has characteristics associated with AI-generated content at a level worth examining more closely." Those are very different statements. Conflating them produces bad outcomes, and in academic integrity cases, "bad outcome" means a student's record.

What instructors know that no detector does

There's something instructors have that no tool has: the baseline. If a student who's been writing at a B-minus level all semester submits an A-plus essay, that discrepancy is interesting — regardless of what any detector returns. If the essay doesn't reflect the vocabulary, framings, or ideas the student used in class discussion, that's interesting too. An instructor who's read 30 essays from the same student over a semester has more useful signal than any classifier.

The impulse to outsource the judgment is understandable. Grading is exhausting, class sizes are large, and a number feels cleaner than a qualitative call. But the number isn't more reliable than the call. It's just easier to point to in a hearing — which is a very different thing.

The signal that actually holds up: evidence of struggle

When I think about what genuinely separates student writing from AI output in practice — not in theory — it comes down to what I'd call the evidence of struggle. Real student writing has friction. Ideas that almost land. Arguments that start somewhere and get corrected mid-paragraph. A sentence that should work but doesn't, followed by a restatement that almost works. That roughness is evidence of a mind in the process of working through something.

AI-generated academic writing is frictionless. Not because it's better — often it's shallower, thinner on actual reasoning — but because it carries no marks of real-time problem-solving. The structure is too clean. The transitions are too smooth. Every claim is followed by a perfectly weighted counterargument. That's not how people think through difficult ideas. It's how a language model distributes probability across tokens.

What better practice looks like

Some schools have started requiring draft submissions alongside final work, with a written process note explaining how the thinking developed. This is harder for AI-assisted work to fake convincingly — the model can generate a draft and a final, but the student has to articulate the evolution of their own reasoning, and that articulation is revealing.

Others have moved toward brief oral follow-ups on major assignments. Not a 20-minute academic defense — just a five-minute conversation where the instructor asks two or three questions about the argument. A student who wrote the essay can generally answer. A student who didn't struggles, usually visibly. No detector required, and the evidence is directly observable rather than probabilistic.

These approaches take more instructor time. That's real, and it's a legitimate constraint at scale. But they're more accurate, more defensible, and more consistent with what writing assignments are actually trying to evaluate — whether the student can work through a problem, not whether they can produce a particular text artifact.

Where I land

I think AI detection in education is worth doing. Not as a surveillance system, not as an automated adjudicator — as a triage tool that helps instructors ask better questions. A breakdown showing high structural uniformity and low opinion drift doesn't tell you a student cheated. It tells you something worth investigating.

Schools that have adopted "score above X equals academic dishonesty charge" are misusing probabilistic instruments in ways that are producing real injustices. That's on the schools, not the tools. But it's worth saying clearly, because the tools will keep being misused until the people deploying them understand what they're actually measuring.

If you work in education and want to use detection responsibly, understanding what these tools actually measure is a reasonable place to start. And running a few sample submissions through Content Trace to see what a category breakdown looks like — before building any policy around a single-score tool — is worth the ten minutes.

Frequently asked questions

Are AI detectors reliable enough for academic integrity cases?

As supporting evidence that prompts further investigation, yes. As standalone proof of cheating, no. Every major AI detection tool carries false positive rates that would be unacceptably high if used as sole evidence in disciplinary proceedings. Turnitin, GPTZero, and Copyleaks have all said as much explicitly in their own documentation.

Why do non-native English speakers get flagged more often?

Perplexity-based detectors flag text that uses predictable vocabulary and simple sentence structures — because those properties overlap statistically with how language models write. Non-native speakers writing carefully to avoid errors tend to produce text with exactly those properties. The Stanford study by Liang et al. (2023) documented this specifically. The bias is structural, not intentional.

What's the best alternative to using a detector as a verdict tool?

Oral follow-up on flagged submissions. Draft submissions with process notes. In-class writing samples to establish a stylistic baseline. Any of these provide more defensible evidence than a probabilistic score, and all of them put the judgment back in the hands of the person with the most context — the instructor.

Should students be told when their work will be run through a detector?

Yes. Transparency about what tools are in use and what role they play in assessment is a basic expectation. Students who know their work may be analyzed are also more likely to write authentically, which serves everyone's interests.

What's the most reliable manual indicator that a student used AI?

The absence of their voice and their specific knowledge. If an essay makes no reference to anything discussed in class, uses none of the course-specific framing, and argues a position the student has never articulated in discussion — that discrepancy is more diagnostic than any score. See the post on why AI writing sounds different for the specific tells to look for.