Extends WildChat
WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training
ICML 2025 · First arXiv submission
Uses WildChat prompts to compare responses from more than 50 open-weight model checkpoints and study the choice of synthetic-data generator. The authors also construct Re-Wild, a supervised fine-tuning mixture, and test it with Llama-3.1 8B.
Data used
A synthetic-response extension of WildChat-1M, retaining its user prompts and adding responses from open-weight models.
Findings
Re-Wild outperforms the Tulu-3 SFT mixture on the paper’s aggregate evaluations using 40% as many training samples.
Section 2: WildChat-1M prompts and 54 model checkpoints. Section 3: SFT mixtures and evaluations. Read the source.
Dates and versions
First arXiv version. ArXiv version history.
ICML 2025 proceedings paper, PMLR volume 267.
Cite this paper
Benjamin Feuer, Chinmay Hegde. 2025. WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training. Proceedings of the 42nd International Conference on Machine Learning.
Show BibTeX
@inproceedings{feuer2025wildchat50m,
title = {{WildChat-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training}},
author = {Feuer, Benjamin and Hegde, Chinmay},
year = {2025},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
publisher = {PMLR},
volume = {267},
pages = {17100--17130},
url = {https://proceedings.mlr.press/v267/feuer25a.html}
}