Uses WildChat and WildBench
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
arXiv preprint · First arXiv submission
Tests whether learning a user simulator from real conversations helps train assistants that work better with people. The study fine-tunes simulators on WildChat user utterances, trains assistants through simulated interaction, and evaluates them on WildBench and in a study with 283 participants.
Data used
WildChat-1M provides human utterances for user-simulator training; WildChat-4.8M and WildBench support evaluation and tests of generalization.
Findings
The assistant trained with the learned user simulator wins 58% of comparisons against the initial assistant in the human study; training with a role-playing model yields 51%.
Sections 2–3 and Appendix C: WildChat preprocessing and simulator training. Section 5: human study. Read the source.
Dates and versions
First arXiv version. ArXiv version history.
arXiv preprint.
Cite this paper
Joseph Suh, Ayush Raj, Minwoo Kang, Serina Chang. 2026. Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants. arXiv preprint.
Show BibTeX
@misc{suh2026utilityusersimulators,
title = {{Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants}},
author = {Suh, Joseph and Raj, Ayush and Kang, Minwoo and Chang, Serina},
year = {2026},
doi = {10.48550/arXiv.2605.09808},
eprint = {2605.09808},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.09808}
}