{
  "id": "long-form-evaluation",
  "title": "如何评估长文本的结构",
  "eyebrow": "篇章连贯性、共指与叙事评估",
  "description": "潜在空间中的模型检验如何超越词元层面的流畅度，评估生成文本的篇章结构、实体行为与主题关系。",
  "lead": "一段话读起来流畅，并不代表整篇文档结构连贯。Yuntian Deng、Volodymyr Kuleshov 和 Alexander M. Rush 的 Model Criticism for Long-Form Text Generation 提出了一个框架：用显式的潜变量检验模型比较真实文本与生成文本的高层结构。论文于 2022 年 10 月 16 日首次发布，发表于 EMNLP 2022，将长文本评估转化为一个明确的统计问题：生成器是否重现了参考文本中的结构分布？",
  "original_papers": [
    "model-criticism"
  ],
  "terms": [
    "长文本生成评估",
    "故事生成评估",
    "篇章连贯性",
    "实体一致性",
    "潜在空间中的模型检验（model criticism）",
    "潜在困惑度（latent perplexity）"
  ],
  "sections": [
    {
      "id": "fluency-and-structure",
      "title": "区分语言流畅与结构连贯",
      "paragraphs": [
        "设想一篇文章先介绍方法、再展示结果，随后却转回一段无关的引言。每句话的语法都正确，章节顺序仍然不合常理。叙事文本也有类似问题：实体突然消失、前后不一致地重新出现，或者在长篇内容中呈现不合理的变化模式。",
        "词元层面的困惑度衡量模型预测词元的能力，却不直接衡量章节转移、共指结构或主题关系。论文用生成过程已知的合成数据说明了这一差别：模型即使擅长预测下一个词元，仍然会对底层潜在状态的转移拟合不佳。"
      ],
      "citations": [
        "model-criticism"
      ]
    },
    {
      "id": "latent-critic",
      "title": "选择能够表达待测性质的检验模型",
      "paragraphs": [
        "这一框架引入一个检验模型（critic），显式建模潜在结构与观测文本的生成过程。它的后验把文档映射成潜变量上的分布。随后，对生成样本和参考数据使用同一个检验模型，比较两者推断出的结构。",
        "一种诊断量是：在检验模型的先验分布下，推断出的潜变量所对应的负对数概率的期望。汇总后得到潜在交叉熵，实验中将其表示为潜在困惑度。这样，被测性质就变得明确：评估章节顺序与评估实体复用需要不同的潜变量模型。检验模型给出关于结构的假设，它与被评估的语言模型是两个不同的模型。"
      ],
      "citations": [
        "model-criticism"
      ]
    },
    {
      "id": "three-properties",
      "title": "三个关于结构的问题",
      "paragraphs": [
        "篇章连贯性关注章节是否遵循真实文档中的转移模式。实验考察了科学论文与维基百科文本，包括提供和不提供显式章节标题提示的生成设置。共指关注生成文本是否重现实体指称的模式。主题性关注主题之间的关系是否与参考文档一致。",
        "这些诊断分别回答不同的问题。论文发现，所评估的 Transformer 语言模型能够捕捉主题相关性，却难以掌握篇章连贯性和共指结构。单一的整体质量分数会掩盖这种差别。该结论对应论文中的模型与检验器；评估新的生成器，需要收集它的输出并重新进行结构比较。"
      ],
      "citations": [
        "model-criticism"
      ]
    },
    {
      "id": "evaluation-landscape",
      "title": "模型检验与故事评估指标如何配合",
      "paragraphs": [
        "OpenMEVA 评估的是评估器本身。它用受控扰动检验自动指标能否识别连贯性错误，以及在不改变语义的变换下能否保持稳定。潜在空间中的模型检验则研究生成器学到了哪些结构分布。两者回答不同的实验问题，能够在同一套评估中配合使用。",
        "Dingyi Yang 和 Qin Jin 的故事评估综述整理了连贯性、人物发展和趣味性等更广泛的标准。针对章节转移的检验器并不衡量所有这些标准。人工评估或 LLM 评判器回答读者关心的问题；明确指定结构的检验模型则提供可复现、针对具体性质的诊断。"
      ],
      "citations": [
        "openmeva",
        "story-evaluation-survey",
        "model-criticism"
      ]
    },
    {
      "id": "evaluation-recipe",
      "title": "设计长文本评估",
      "paragraphs": [
        "先明确要检测什么错误，再定义对应的潜在表示与参考语料。拟合或指定检验模型，检查它在真实文本和已知结构错误上的表现，然后以相同方式处理参考样本和生成样本。明确报告生成条件、文档长度和样本数量，让比较回答预定的问题。",
        "分别报告词元层面与结构层面的指标。检查异常潜在分数背后的文本，也检查检验模型自身的推断错误。分布不匹配指出需要诊断的方向，并不意味着每一个不同寻常的故事都不好。原论文和开源代码分别实现了篇章连贯性、共指和主题性的诊断。"
      ],
      "citations": [
        "model-criticism",
        "openmeva"
      ]
    }
  ],
  "related_guides": [
    "probabilistic-attention",
    "structured-decoding"
  ],
  "demo": "structure",
  "reading_order": [
    "model-criticism",
    "openmeva",
    "story-evaluation-survey"
  ],
  "updated_at": "2026-11-11",
  "url": "https://yuntiandeng.com/research/zh/long-form-evaluation/",
  "papers": [
    {
      "id": "model-criticism",
      "title": "Model Criticism for Long-Form Text Generation",
      "authors": [
        {
          "given": "Yuntian",
          "family": "Deng"
        },
        {
          "given": "Volodymyr",
          "family": "Kuleshov"
        },
        {
          "given": "Alexander M.",
          "family": "Rush"
        }
      ],
      "year": 2022,
      "venue": "EMNLP 2022",
      "paper_url": "https://aclanthology.org/2022.emnlp-main.815/",
      "pdf_url": "https://aclanthology.org/2022.emnlp-main.815.pdf",
      "code_url": "https://github.com/da03/criticize_text_generation",
      "doi": "10.18653/v1/2022.emnlp-main.815",
      "summary": "通过在显式生成式检验模型定义的潜在空间中比较真实数据与生成数据，评估文本的高层结构。实验研究篇章连贯性、共指和主题性。",
      "connection": "诊断生成文本结构分布的原始框架，将结构评估与词元层面的流畅度分开。",
      "bibtex_type": "inproceedings",
      "citation_key": "deng-etal-2022-model",
      "source_urls": [
        "https://aclanthology.org/2022.emnlp-main.815/",
        "https://arxiv.org/html/2210.08444v1"
      ],
      "arxiv_id": "2210.08444",
      "arxiv_first_posted": "2022-10-16",
      "abstract": "Language models have demonstrated the ability to generate highly fluent text; however, it remains unclear whether their output retains coherent high-level structure (e.g., story progression). Here, we propose to apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of the generated text. Model criticism compares the distributions between real and generated data in a latent space obtained according to an assumptive generative process. Different generative processes identify specific failure modes of the underlying model. We perform experiments on three representative aspects of high-level discourse—coherence, coreference, and topicality—and find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.",
      "abstract_source": "https://arxiv.org/html/2210.08444v1",
      "updated_at": "2026-11-11",
      "description_language": "zh-Hans",
      "abstract_language": "en"
    },
    {
      "id": "openmeva",
      "title": "OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics",
      "authors": [
        {
          "given": "Jian",
          "family": "Guan"
        },
        {
          "given": "Zhexin",
          "family": "Zhang"
        },
        {
          "given": "Zhuoer",
          "family": "Feng"
        },
        {
          "given": "Zitao",
          "family": "Liu"
        },
        {
          "given": "Wenbiao",
          "family": "Ding"
        },
        {
          "given": "Xiaoxi",
          "family": "Mao"
        },
        {
          "given": "Changjie",
          "family": "Fan"
        },
        {
          "given": "Minlie",
          "family": "Huang"
        }
      ],
      "year": 2021,
      "venue": "ACL-IJCNLP 2021",
      "paper_url": "https://aclanthology.org/2021.acl-long.500/",
      "pdf_url": "https://aclanthology.org/2021.acl-long.500.pdf",
      "doi": "10.18653/v1/2021.acl-long.500",
      "summary": "检验自动故事评估指标的基准，涵盖与人工判断的相关性、泛化、稳健性，以及对不同连贯性错误的敏感程度。",
      "connection": "检验评估器能否发现已知的故事缺陷，与在潜在空间中诊断生成器输出分布的方法互补。",
      "bibtex_type": "inproceedings",
      "citation_key": "guan-etal-2021-openmeva",
      "source_urls": [
        "https://aclanthology.org/2021.acl-long.500/"
      ],
      "updated_at": "2026-11-11",
      "description_language": "zh-Hans",
      "abstract_language": "en"
    },
    {
      "id": "story-evaluation-survey",
      "title": "What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation",
      "authors": [
        {
          "given": "Dingyi",
          "family": "Yang"
        },
        {
          "given": "Qin",
          "family": "Jin"
        }
      ],
      "year": 2024,
      "venue": "arXiv",
      "paper_url": "https://arxiv.org/abs/2408.14622",
      "pdf_url": "https://arxiv.org/pdf/2408.14622",
      "doi": "10.48550/arXiv.2408.14622",
      "summary": "综述故事质量标准、人工评估、自动指标和评估基准。",
      "connection": "将结构诊断放在连贯性、人物发展与叙事质量的更广泛评估背景中。",
      "bibtex_type": "article",
      "citation_key": "yang2024goodstory",
      "source_urls": [
        "https://arxiv.org/abs/2408.14622",
        "https://arxiv.org/pdf/2408.14622"
      ],
      "arxiv_id": "2408.14622",
      "arxiv_first_posted": "2024-08-26",
      "publication_type": "preprint",
      "container_title": "arXiv",
      "updated_at": "2026-11-11",
      "description_language": "zh-Hans",
      "abstract_language": "en"
    }
  ]
}