An AUROC score does not set the false-positive rate

A cutoff turns a score
into a flag

Different cutoffs trade missed AI text against wrongly flagged human text

AUROC summarizes ranking across cutoffs. It does not select an operating point.

The classifier combines many small decision trees

300

trees

6

maximum depth

3

minimum leaf size

97 measurements, balanced class weights, fixed seed

Main random forest configuration

A decision path can use only formatting tests

1

Markdown bold present?
Yes

2

Double-backtick inline code present?
No

Result

Tree predicts LLM

One actual path in the illustrative depth-3 tree. Its AUROC is 0.7497; the main forest is a different model.

The 97 measurements cover five surface families

Family
Example
Count
Prose typography
'   ’   ...   …
22
Markdown
**bold**   __bold__
25
Math markup
$x$   \(x\)
24
Code formatting
Tabs   spaces
20
Whitespace
Blank lines
6
Features include rates, densities and indicators computed from raw text

The main reference-model comparators

Mean AUROC across the 17 model sources0.5Chance0.60.70.80.91.0PerfectSurfacePrint, raw0.983Fast-DetectGPT0.950Lastde++0.942DNA-DetectLLM0.942
Same held-out texts. SurfacePrint learns from labels; these baselines are training-free detectors.

The source pool and detector sample differ

Source pool, per domain

2,000 train / 500 validation / 1,000 test prompts

Main detector’s training sample

8,000 human posts + 470 outputs per AI model

17 × 470 = 7,990 generated examples

Four source domains. Held-out test examples do not enter detector training.

SP-WS standardizes selected whitespace conventions

Line breaksUse one line-break encoding
Spaces at line endsRemove them
Repeated blank linesKeep one blank line
Final line breakRemove it

Punctuation and markup remain in the input

SP-WS is a separately trained variant. Some layout information remains.

A next evaluation would test real document handling

Representative documents
Human and generated text from the intended setting

Real processing steps
Include ordinary editing, copying and extraction

Held-out error measurement
Report false positives and detection at a fixed cutoff

Proposed follow-up evaluation, not a completed experiment