Coherence, coreference and narrative evaluation
A fluent paragraph does not establish a coherent document. Model Criticism for Long-Form Text Generation, by Yuntian Deng, Volodymyr Kuleshov, and Alexander M. Rush, introduced a framework for comparing the high-level structures of real and generated text through explicit latent-variable critics. First released on October 16, 2022 and published at EMNLP 2022, it connects long-form evaluation to a concrete statistical question: does a generator reproduce the distribution of structures found in reference texts?
Original research
Model Criticism for Long-Form Text Generation
Yuntian Deng, Volodymyr Kuleshov, Alexander M. Rush · EMNLP 2022
Work through an example
Illustrative section sequences, not evaluation scores from the paper.
The section order progresses through the document.
A critic maps text to a structural representation. The paper compares distributions of these representations over real and generated corpora.
Consider an article that introduces a method, presents results, and then returns to an unrelated introduction. Each sentence can be grammatical while the section sequence is structurally unusual. A narrative has related problems when entities disappear, reappear inconsistently, or follow implausible patterns across a long passage.
Word-level perplexity measures how well a model predicts tokens. It does not directly measure section transitions, coreference structure, or topical relationships. The paper demonstrates this distinction with synthetic data whose generative process is known: strong next-token prediction coexists with a poor fit to the underlying latent transitions.
The framework introduces a critic with a generative model of latent structure and observed text. Its posterior maps a document into a distribution over latent variables. Criticism then compares the structures inferred from generated samples with those inferred from reference data, using the same critic for both.
One diagnostic is the expected negative log probability of inferred latent variables under the critic’s prior. Aggregating it produces a latent cross-entropy, reported as latent perplexity in the experiments. This makes the measured property explicit: an evaluator of section order and an evaluator of entity reuse need different latent models. The critic supplies an assumption about structure; it is not the language model under evaluation.
Discourse coherence asks whether sections follow the transitions found in real documents. The experiments examine scientific articles and Wikipedia, including generation with and without explicit section-title cues. Coreference asks whether generated text reproduces the patterns of references to entities. Topicality asks whether topic relationships match reference documents.
These are distinct diagnoses. The paper finds that its transformer language models capture topical correlations while struggling with coherence and coreference. A single overall quality score would hide this difference. The conclusion concerns the evaluated models and critics; testing a new generator requires collecting its outputs and running the structural comparison again.
OpenMEVA evaluates the evaluators. Its controlled perturbations test whether automatic metrics respond to errors in coherence and remain stable under meaning-preserving changes. Latent-space model criticism instead asks which structural distributions a generator has learned. The two approaches answer different experimental questions and fit together in an evaluation suite.
The story-evaluation survey by Dingyi Yang and Qin Jin organizes broader criteria such as coherence, character development, and interestingness. A critic for section transitions does not measure all these criteria. Human assessment or an LLM judge answers reader-facing questions; a specified structural critic supplies a reproducible diagnostic tied to a particular property.
Sources: OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation MetricsWhat Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story EvaluationModel Criticism for Long-Form Text Generation
Start with the failure you want to detect, then define its latent representation and reference corpus. Fit or specify the critic, check its behavior on real texts and known structural errors, and apply it identically to reference and generated samples. Keep generation conditions, document lengths, and sample counts visible so that the comparison answers the intended question.
Report word-level and structural measurements separately. Inspect examples behind unusual latent scores, including errors in the critic’s own inference. Distributional mismatch identifies a diagnostic target; it does not establish that every unusual individual story is bad. The original paper and released code provide concrete starting points for coherence, coreference, and topicality rather than a single universal quality metric.
Sources: Model Criticism for Long-Form Text GenerationOpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics
EMNLP 2022
A framework for evaluating high-level text structure by comparing real and generated data in latent spaces defined by explicit generative critics. Experiments study discourse coherence, coreference, and topicality.
The original method behind this guide: diagnostic evaluation of structural distributions, separate from word-level fluency.
ACL-IJCNLP 2021
A benchmark for testing automatic story metrics, including human correlation, generalization, robustness, and sensitivity to different coherence errors.
Tests whether an evaluator detects known story defects; complements latent-space criticism of a generator’s output distribution.
arXiv
A survey of story-quality criteria, human evaluation, automatic metrics, and evaluation benchmarks.
Places structural criticism within the broader evaluation of coherence, character development, and narrative quality.