{"id":"wildbench","title":"WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild","authors":[{"given":"Bill Yuchen","family":"Lin"},{"given":"Yuntian","family":"Deng"},{"given":"Khyathi","family":"Chandu"},{"given":"Faeze","family":"Brahman"},{"given":"Abhilasha","family":"Ravichander"},{"given":"Valentina","family":"Pyatkin"},{"given":"Nouha","family":"Dziri"},{"given":"Ronan","family":"Le Bras"},{"given":"Yejin","family":"Choi"}],"first_public":"2024-06-07","date_note":"First arXiv version; the paper also describes benchmark releases in March and May 2024.","date_source":"https://arxiv.org/abs/2406.04770","year":2025,"venue":"ICLR 2025","paper_url":"https://openreview.net/forum?id=MKEHCx25xp","arxiv":"2406.04770","code_url":"https://github.com/allenai/WildBench","category":"evaluation","relationship":"uses","dataset_version":"WildBench V2: 1,024 tasks curated from WildChat collection data and held out of the public WildChat training corpus.","summary":"Turns challenging user requests from the WildChat collection into an evaluation set with task-specific checklists. The selection process combines language and quality filters, model difficulty estimates, and manual review, while reserving selected conversations from the public WildChat release.","finding":"For the top-ranked models evaluated in the paper, WB-Reward has a Pearson correlation of 0.98 with Chatbot Arena's human-voted Elo ratings.","evidence_url":"https://arxiv.org/html/2406.04770v2","evidence_note":"Section 2.1 specifies selection and held-out data; Sections 3–4 define the metrics and compare model rankings.","bibtex_key":"lin2024wildbench","record_url":"https://yuntiandeng.com/wildchat/papers/wildbench/","relationship_label":"Uses WildChat","citation_title":"WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild","citation_type":"article-journal","bibtex_type":"misc","citation_url":"https://arxiv.org/abs/2406.04770","citation_year":2024,"citation_note":"Cites the arXiv preprint with its original 2024 preprint year. The paper later appeared at ICLR 2025.","doi":"10.48550/arXiv.2406.04770"}