Research talk ·
SurfacePrint
Typographic Fingerprints of LLMs
Small formatting choices carry information about how text was produced. SurfacePrint asks how much that information can help distinguish human and language-model text.
The PDF includes all 14 main slides and 8 additional slides. The slides use a concise spoken-talk format; the explanation below supplies context for reading them independently.
What the study shows
The opening example shows how the same words can have different surface forms, motivating the measurements used by the classifier; it is an illustration, not an experimental result.
SurfacePrint represents a text with 97 measurements of its formatting: punctuation, Markdown, math markup, code formatting and whitespace. A random forest, an ensemble of 300 small decision trees, learns from this representation to distinguish human and generated text. Scoring a new passage requires no language-model calls.
On held-out texts from 17 model sources represented in training, SurfacePrint reaches a mean AUROC of 0.983, compared with 0.950 for Fast-DetectGPT. Both use the same test texts, but SurfacePrint learns from labeled examples; Fast-DetectGPT is a training-free detector.
A separately trained variant reaches 0.889 after selected whitespace cleanup. This comparison measures two trained variants; it does not test whether the original detector withstands edits.
Instruction-tuned outputs are easier to distinguish from human text in four of five tested model families. Each condition uses a separately trained detector, and prompt templates also differ, so this comparison does not isolate the effect of weight updates.
How to read the results
AUROC measures how well scores rank generated text above human text: 0.5 is chance ranking and 1.0 is perfect. It is neither accuracy at a chosen cutoff nor a document’s probability of AI authorship.
AI writing styles leave revealing fingerprints. Here, a fingerprint means a statistical pattern, not a unique identity. Transfer to unseen models is uneven, so a strong result on known model sources does not guarantee the same performance on a new model.
When a new model family appears, the practical next step is to collect labeled examples of its outputs alongside human writing, retrain SurfacePrint to learn its formatting patterns, and use the updated detector to detect AI-generated text. This is a recommendation based on uneven transfer and evolving conventions; the study does not test a continuing retraining loop or establish that updating always restores performance. An updated detector needs evaluation on held-out examples.
Formatting is editable, and these experiments address non-adversarial detection. Practical use needs measured false-positive rates on representative documents; the score should not be sole evidence of authorship.
The manuscript is not yet publicly available.