Why Your AI Detector Score Keeps Changing (And What to Do About It)

When OpenAI released its own AI text classifier in January 2023, it shipped with an unusual warning: "The classifier is very unreliable on short texts (below 1,000 characters)." It correctly flagged 26% of AI-written text and wrongly flagged human text 9% of the time. Six months later, OpenAI took it down, citing its "low rate of accuracy."

That was the company that built the model, trying to detect its own output. So when a writer pastes the same blog post into four detectors and gets four very different numbers, nothing has broken. The frustration is real, especially when one tool says 12% and another says 88% on a paragraph the writer typed by hand. But the variation has causes, and once you know them, a changing score becomes much less alarming and a lot more useful.

Why different tools give different scores

Different detectors measure different things, and that explains most of it. A tool built mainly on word predictability will disagree with a tool that weighs how a text argues, because the same text can look ordinary on one dimension and unusual on the other. Each tool reports on a different property of the same text.

Then there's the reference model. Predictability is always measured against some language model: how surprising are these words, given what that model would expect? Tools use different reference models. Text that looks expected to one can look surprising to another, and text written by a model the tool never saw in training is the hardest case of all.

Training data matters too. Every classifier learns from labeled examples of human and AI text. Which years, which models, which genres: all of that shapes what it recognizes. Two tools trained on different sets will split on edge cases, and most interesting writing is an edge case of some kind.

Here's a surprising one from the largest independent test of the period. Weber-Wulff and seven co-authors tested 14 tools on 54 documents. Human-written text was classified correctly 96% of the time on average. Human text that had been machine-translated into English scored about 20% lower. Nothing about who wrote it changed. Only the route the words took into English.

Finally, thresholds differ. One tool's 60% is not another tool's 60%. Each vendor decides where its scale bends, and few publish how. Comparing raw numbers across tools is a bit like comparing shoe sizes across countries without the chart.

Why scores vary across runs of the same tool

This surprises people more. Run the same text through the same detector twice and get 71%, then 68%. If the input didn't change, why did the output?

A few reasons. Some tools use language models with random sampling switched on, so the same input can produce slightly different results from one call to the next. Some tools split text into chunks, score each one and combine them, and a medium-length text isn't always split the same way. Pasting from a different source can add or drop headings, lists and line breaks, which changes the chunks too. And some vendors update their classifiers without telling anyone, so the tool you used last Tuesday may not be the tool you're using today.

A few points of drift between runs is normal. Swings of ten points or more on identical text point to a tool that isn't stable, and that's worth knowing before anyone makes a decision based on its number.

The text length problem

Every detector gets less reliable as text gets shorter. Most signals need several instances before a pattern means anything. One sentence can be unusual whether a person or a model wrote it. Several paragraphs give a tool enough repetitions to tell a habit from an accident.

The vendors say this themselves, in small print. Turnitin needs at least 300 words of prose to produce a result. It also marks scores between 1% and 20% with an asterisk, because its testing found "a higher incidence of false positives" in that range. Few people who see that asterisk know what it means.

The awkward part: much of what people most want to check is short. Social posts, email copy, ad text. A tweet that scores 85% AI tells you very little. A long article with the same patterns showing up section after section tells you a lot more.

Score instability is information

When a text scores very differently across tools, the split says something about the text, as well as about the tools.

Unedited AI output tends to score high across well-built tools, because several independent signals point the same way. Clearly human writing tends to score low for the same reason. Text that sends tools in different directions usually sits in mixed territory: an AI-assisted draft where the writer added real material in some sections and not others, formal human writing that shares surface patterns with models, or a document stitched from several sources.

The obvious fix is to run four tools and average the results. It doesn't work, because the tools aren't measuring the same thing, so the average describes nothing in particular. A better habit is to look for agreement. If several tools with different methods all point at the same section, that section probably needs work. If one tool out of four disagrees, that's one tool.

How to get more consistent results

Check longer text

If a piece is under a few hundred words, check it together with the content around it, or accept that the number is soft. More text gives more stable readings.

Read the breakdown before the headline number

A single number hides what drives it. A breakdown shows which qualities of the writing are pulling the result, and one odd sentence moves a section less than it moves a total. Content Trace explains its results with 32 signals in 8 sections for exactly this reason: a writer can see which part of the draft to edit next.

Treat the middle as a to-do list

A mid-range score means features point both ways. That's a reason to look at the weakest sections and improve them, and never a reason to accuse anyone. The goal of checking an AI-assisted draft is a better draft, one with real examples and a clear position, that readers finish.

Frequently asked questions

For the mechanics behind these tools, see How AI Text Detection Actually Works. Can AI Detectors Be Fooled? covers why chasing a lower score is the wrong goal. And How to Read a Detection Report walks through a breakdown section by section.