OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

Method and contribution

A benchmark for testing automatic story metrics, including human correlation, generalization, robustness, and sensitivity to different coherence errors.

Tests whether an evaluator detects known story defects; complements latent-space criticism of a generator’s output distribution.

Read in context

Evaluating the structure of long-form text →

Citation

Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, Minlie Huang. OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics. ACL-IJCNLP 2021.

@inproceedings{guan-etal-2021-openmeva,
  title = {{OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics}},
  author = {Guan, Jian and Zhang, Zhexin and Feng, Zhuoer and Liu, Zitao and Ding, Wenbiao and Mao, Xiaoxi and Fan, Changjie and Huang, Minlie},
  year = {2021},
  booktitle = {ACL-IJCNLP 2021},
  url = {https://aclanthology.org/2021.acl-long.500/},
  doi = {10.18653/v1/2021.acl-long.500}
}