Uses WildChat
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
ICLR 2025 · First arXiv submission
Turns challenging user requests from WildChat into an evaluation set with task-specific checklists. The selection process combines language and quality filters, model difficulty estimates, and manual review. Selected conversations are held out of the public WildChat release.
Data used
WildBench V2: 1,024 tasks curated from WildChat collection data and held out of the public WildChat training corpus.
Findings
For the top-ranked models evaluated in the paper, WB-Reward has a Pearson correlation of 0.98 with Chatbot Arena's human-voted Elo ratings.
Section 2.1: selection and held-out data. Sections 3–4: metrics and model rankings. Read the source.
Dates and versions
First arXiv version; the paper also describes benchmark releases in March and May 2024. ArXiv version history.
2024 arXiv preprint; published at ICLR 2025.
Cite this paper
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, Yejin Choi. 2024. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. arXiv preprint.
Show BibTeX
@misc{lin2024wildbench,
title = {{WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild}},
author = {Lin, Bill Yuchen and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Le Bras, Ronan and Choi, Yejin},
year = {2024},
doi = {10.48550/arXiv.2406.04770},
eprint = {2406.04770},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2406.04770}
}