Writing Under Surveillance: The Problem with AI Detection

Be it on Reddit or Discord, CUNY students often express anger and anxiety at the prospect of being falsely flagged by AI detection software, their professors entrusting algorithmic systems over the word and testimony of their students.

Plagiarism detection giants like Turnitin have long shaped how students learn to write and under what conditions they see themselves as writers. More recently, the plot has thickened, with these platforms marketing AI detection services that claim to distinguish human-written text from machine-generated output. Now, international students report accusations of academic dishonesty for syntax that is "too clean" while others testify to misspelling words or introducing mistakes in their writing to beat the algorithm and evade detection, even when every word is their own.

When students feel the need to perform their humanity for an audience of algorithms, something has gone wrong. These students write not for their human peers but for automated systems, seldom even understood by professors who mandate them or administrators who procure them. From wrongful accusations to self-policing practices, the harms of these tools are diverse and diffuse and often fall hardest on working-class and non-native English-speaking students like so many enrolled across the CUNY system.

To grasp the problem with AI detection, let's consider what a prominent service claims to do. GPTZero reports use of two metrics: one is perplexity, a measure of how well a LLM would predict each successive word in a passage; another is burstiness, used to score variation in the sentence-level rhythm and structure of student writing (Tian). Of course, this depends on the assumption that human writers intuitively exhibit syntactic diversity, and mix long and short sentences together by default, while AI models lean toward consistent, flat tempos at the sentence- and paragraph-level ("AI Detectors"; Galczynski).

This premise is incredibly shaky; it's also constitutive of detection failures at scale. One of the most comprehensive accounts in the field tested twelve publicly available tools alongside Turnitin and PlagiarismCheck, reporting they were "neither accurate nor reliable," and exhibited a systematic bias toward classifying AI-generated text as human-written (Weber-Wulff et al.). More broadly, it is also worth noting that even a "low" 1% false positive rate across 22.35 million first-year college essays amounts to 223,500 essays falsely flagged in a single year (Hirsch).

Those affected are hardly edge cases. The MLA-CCCC Joint Task Force on Writing and AI put it plainly when it warned AI detection tools enable "false accusations" that "may disproportionately affect marginalized groups." Sure enough Black students face disproportionately higher rates of AI-detection accusations than their white peers (Madden et al.). Students identifying as disabled are also reported more likely to receive a false positive for submitted writing that departs from the narrowed standards these tools inscribe as human (Hirsch). Lastly, one study of seven widely used detectors found these services flagged as AI-generated 61.22% of TOEFL essays by non-native English speakers, and, across all seven detectors, 89 of 91 essays were flagged by at least one of the tools sampled in the study (Liang et al.).

The failings of AI detection software matter all the more because of how these services rely on a construct of authenticity narrowed to features the algorithm can count, weight, and score — and that legibility is not evenly distributed, especially among student groups as diverse as those at CUNY.

The "human" criteria these systems reify is less an inherent quality of submitted writing than a dynamic classification scheme for evaluating prose, as if its standards were always already there. In return students are left to bear the burden of an automated specter casting doubt over different kinds of writing they might otherwise use to express themselves.

Part of the issue lies in the "problems" these tools presume of us and the solutions they claim to provide alongside them. At the same time, detection treats writing as a process under surveillance, invites suspicion in place of trust and audience, and recasts assessment as a form of automated verification rather than a responsive, context-aware teaching practice.

For all their alleged sophistication, AI detectors are still trying to sort language into patterns that do not hold consistently across LLMs and that are easily disrupted by simple obfuscation strategies (Weber-Wulff et al.). Their unreliability, and the anxiety they produce, is therefore hardly surprising. The larger problem is that they invite us to accept a hidden standard of human writing that is at once opaque and flattening, one that treats authentic prose as a fixed object or end product. As large language models get better and better at simulating human communication, the goal posts will continue to shift as the line between human- and machine-generated text narrows by the day.

Building trust in the classroom is hard work, and AI detectors undermine that effort at every turn, like a wolf in sheep's clothing, a problem dressed in its own solution. We and our students are better served when we take time to slow down, start small, and treat reading and writing as social projects still in the making. And for how fraught authorship is these days, it'd do us well to leave the problem of AI detection at the door.