How Machines Write, and Why It Leaves Fingerprints

Every detector in this course, from a classroom tool to a research paper, rests on one bet. The bet is that the way a language model picks words leaves a pattern, and the way people pick words leaves a different one. To judge any detector, you need to know where that pattern comes from. That means knowing, at a mechanical level, how a machine writes a sentence.

The course on how AI works covers the full machinery in How large language models work. This lesson takes only the parts that matter for detection.

Tokens: the units of machine writing

A language model does not see letters or whole words. It sees tokens, pieces of text drawn from a fixed vocabulary. Common words are a single token. Rare words, names and long numbers get chopped into several. OpenAI's help page gives the rule of thumb for English: one token is about four characters, about three quarters of a word, and 100 tokens is about 75 words. Other languages break down differently, often into more tokens per word.

Detection works on tokens too. When a detector says a passage is "predictable," it means the model behind the detector could guess each token from the ones before it. That is why length rules matter so much. A 40-word comment is roughly 55 tokens, and 55 guesses is a thin sample for any statistical judgment.

Writing is a loop of guesses

Here is the whole act of machine writing. The model takes the text so far, including your prompt, and computes a probability for every token in its vocabulary: how likely each one is to come next. One token is picked. It is attached to the end. The longer text goes back in, and the model computes a fresh list. A 300-word answer is about 400 trips through that loop.

Take the start of a sentence in a county news release: "The commissioners voted to approve the." A model might rate the next token like this (illustrative numbers):

Candidate next token	Probability
budget	0.31
contract	0.18
proposal	0.14
request	0.09
... tens of thousands of others	the remaining 0.28, spread thin
gravel	0.0004

A person writing that sentence knows which item was on the agenda. Maybe it was gravel for a road district. The model knows only what usually follows those words in the text it was trained on. So the model leans toward "budget," and the human picks whatever actually happened. Multiply that difference across hundreds of word choices and you get the fingerprint detectors look for.

Sampling, temperature and why output varies

If a model always took the single most likely token, every answer to the same prompt would be identical. Worse, the text would be dull. In a 2020 paper at the ICLR conference, Ari Holtzman and colleagues showed that decoding methods built to find the most probable text, such as beam search, produce text that is "bland and strangely repetitive." Their charts showed something important for detection: human text, scored by the same model, bounces between likely and unlikely tokens, with much more variance than machine-maximized text.

So real systems sample. They roll weighted dice over the probability list. Two common controls shape that roll:

• Temperature sharpens or flattens the list before sampling. OpenAI's own library documents it this way: "Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic." The allowed range is 0 to 2.
• Top-p, or nucleus sampling, the method Holtzman's team proposed, trims the list to the smallest group of top tokens whose probabilities add up to a set share and samples only from those. OpenAI's library gives the example that 0.1 means only the tokens in the top 10 percent of probability mass are considered.

Both controls cut off the long tail of strange words most of the time. That is good for readability. It also means machine text rarely reaches for the 0.0004 word. Human writers do it constantly, because they are describing a specific road, a specific meeting, a specific grudge.

There is a catch that runs through every later lesson. Sampling settings are a dial, and the person running the model can turn it. Daphne Ippolito and colleagues found in 2020 that the decoding tricks that make text fool human readers also leave statistical oddities that make it easier for software to catch. Changing the settings changes which signals show up. A detector tuned on one setting can miss text made with another.

Instruction tuning and the house style

A raw model trained only to predict the next token on internet text is called a base model. It will continue a forum post, a recipe or a legal brief in whatever voice the opening suggests. Chatbots are base models with extra training on top.

The best-known description of that step is OpenAI's 2022 InstructGPT paper by Long Ouyang and colleagues. People wrote example answers to prompts, the model was trained on those, then people ranked several model answers from best to worst, and the model was trained again to produce answers like the top-ranked ones. That last step is reinforcement learning from human feedback, or RLHF. In their tests, people preferred answers from a 1.3 billion parameter InstructGPT model over answers from the 175 billion parameter GPT-3, a model more than 100 times larger.

That training does more than teach manners. It narrows the range of what the model says.

• Robert Kirk and colleagues reported in 2023 that RLHF "significantly reduces output diversity" compared with the earlier supervised step, across several measures.
• A Carnegie Mellon team led by Alex Reinhart compared human and model writing on the same prompts, using Douglas Biber's set of grammatical and rhetorical features. GPT-4o and Llama variants used present participle clauses at 2 to 5 times the human rate and nominalizations (nouns made from verbs, like "implementation") at 1.5 to 2 times. GPT-4o used the agentless passive at about half the human rate. The gaps were larger for instruction-tuned models than base models, and they persisted from smaller models to larger ones. The study appeared in PNAS in 2025.
• The same study found ChatGPT versions used "camaraderie" and "tapestry" about 150 times more often than humans writing in the same genres, and Llama variants used "unease" 60 to 100 times more often.

Reinhart put it plainly: "LLMs, like ChatGPT and Llama, write a certain way, and they don't necessarily adapt to the writing style." Raters rewarded a certain kind of answer, clear, organized, agreeable and dense with nouns, and the models learned to give that answer every time. That consistency is a house style. Detectors, and experienced readers, learn to recognize it. Lesson 5, The tells, looks at the word-level version; this lesson stays with the statistics.

Perplexity: how surprised is the model?

Perplexity is the oldest number in this field. It was built to grade language models, and detection borrowed it.

The definition, as the Hugging Face documentation states it: perplexity is the exponentiated average negative log-likelihood of a sequence. In plain words, run the text through a model, record the probability the model gave to each token that actually appears, and average how surprised it was. If the model kept expecting exactly the words on the page, perplexity is low. If the words kept surprising it, perplexity is high.

GPTZero's founder, Edward Tian, gave a simple example in a March 2023 explainer. After "Hi there, I am an AI," a model would most likely predict "assistant." That continuation has low perplexity. If the next word were "potato," perplexity would be much higher, and the sentence would more likely have come from a person.

A useful way to read the number: a perplexity of 10 means the model was, on average, about as uncertain as if it were choosing among 10 equally likely tokens at each step. A perplexity of 2 means it was nearly sure every time.

Burstiness: how much does the surprise move?

Perplexity scores a whole passage. Burstiness looks at how that score changes across the passage. GPTZero's explainer describes it as "a measure of how much writing patterns and text perplexities vary over the entire document," and says people "tend to vary" their sentence construction while models "write with a very consistent level of AI-likeness."

Picture five sentences from each of two essays, each sentence scored for perplexity:

Essay	Sentence perplexities	Average	Spread (standard deviation)	Spread as share of average
Steady	12, 14, 11, 13, 12	12.4	1.0	0.08
Bursty	8, 45, 15, 90, 22	36.0	29.7	0.83

The bursty essay has one plain sentence, one wild one, a few in between. That is how people tend to write when they mix a statistic, a quote, a joke and a flat summary. The steady essay keeps the same level of predictability all the way through. Under burstiness scoring, the steady pattern points toward a machine. The numbers in this table are illustrative; the method of comparing spread is the real one.

Sentence length works the same way and is easier to see without software. Count the words in each sentence of a paragraph. Human writers who are not working from a template tend to swing from four words to forty. Chatbot answers often hold to a narrower band. That is a reader's version of burstiness, and lesson 5 covers how weak it is as evidence alone.

Why human writing tends to vary more

The research gives several reasons, and they stack.

People write from facts the model does not have. The name of a road, the year a mill closed, what the clerk said at the counter. Those details are low-probability tokens by definition, because they are specific. A model generating from patterns supplies the typical detail instead.

People do not pick the top word. Sebastian Gehrmann, Hendrik Strobelt and Alexander Rush built a tool called GLTR in 2019 that colors each word by where it ranked in a model's prediction list: green for the top 10, yellow for the top 100, red for the top 1,000, purple for the rest. Generated text came out almost entirely green and yellow. A real scientific abstract showed far more red and purple. Their classifier learned that real text "samples from the tail of the distribution more frequently." In a test with people, showing them GLTR's colors raised their detection rate from 54 percent to 72 percent without other training.

Training rewards one voice. As above, instruction tuning narrows output. Human writers were never ranked against each other by the same panel of raters, and it shows.

Moods, drafts and interruptions. A person writes one paragraph tired and the next angry, revises some sentences five times and leaves others raw. A model produces every sentence with the same procedure.

Where the fingerprint fades

Every reason above is a tendency. Each one fails in known, predictable ways, and those failures are where false accusations come from.

• Formulaic human writing. Lab reports, legal notices, grant boilerplate, five-paragraph essays and meeting minutes are low-perplexity by design. People are taught to write them that way.
• Second-language writing. Writers working in a second language often use common words and safe structures. In the Stanford study by Weixin Liang and colleagues, seven detectors flagged real TOEFL essays as AI-written 61.22 percent of the time on average. When the researchers had ChatGPT enrich the vocabulary of those same essays, perplexity rose and the false positive rate fell to 11.77 percent. When they had it simplify the word choices of essays by U.S. eighth graders to mimic a non-native writer, the average false positive rate jumped from 5.19 percent to 56.65 percent. Notice the oddity in the first test: the TOEFL essays ChatGPT had rewritten were flagged far less often than the originals the students wrote themselves. The detectors were measuring vocabulary range, and that tracks with language background.
• Edited machine text. A person who rewrites a model's draft, adds their own details and cuts the padding raises perplexity and burstiness. So does a paraphrasing tool. Lesson 6, Accuracy and false positives, covers the studies on evasion.
• Different settings and models. A higher temperature, a different model, or a prompt asking for a quirky style all shift the numbers. Perplexity is always measured against one particular scoring model, and a text can look predictable to one model and surprising to another.

That last point is why the field moved on. GPTZero, the tool that made perplexity and burstiness famous, added a note to its own explainer saying that as of autumn 2023 it "no longer uses perplexity and burstiness" for detection, having moved to a deep-learning system. Most commercial detectors now train classifiers on large sets of labeled human and machine text, and the strongest research methods measure probability in more careful ways. Lesson 3, How detectors work, takes those methods one at a time.