Uses WildChat
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
ICLR 2025 · First arXiv submission
Turns challenging user requests from the WildChat collection into an evaluation set with task-specific checklists. The selection process combines language and quality filters, model difficulty estimates, and manual review, while reserving selected conversations from the public WildChat release.
Dataset and scope
WildBench V2: 1,024 tasks curated from WildChat collection data and held out of the public WildChat training corpus.
Selected finding
For the top-ranked models evaluated in the paper, WB-Reward has a Pearson correlation of 0.98 with Chatbot Arena's human-voted Elo ratings.
Section 2.1 specifies selection and held-out data; Sections 3–4 define the metrics and compare model rankings. Read the source.
Dates and versions
First arXiv version; the paper also describes benchmark releases in March and May 2024. ArXiv version history.
Cites the arXiv preprint with its original 2024 preprint year. The paper later appeared at ICLR 2025.
The annotation describes the linked source version. Consult the paper for its full methods, assumptions, and limitations.
Cite this paper
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi. 2024. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. arXiv preprint.
Show BibTeX
@misc{lin2024wildbench,
title = {{WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild}},
author = {Lin, Bill Yuchen and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Le Bras, Ronan and Choi, Yejin},
year = {2024},
doi = {10.48550/arXiv.2406.04770},
eprint = {2406.04770},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2406.04770}
}Reuse this entry
JSON recordFull reading list in MarkdownFull bibliography
Annotations are a reading aid prepared for this hub. The linked authors’ work is the source for research claims. About the selection and corrections.