Extends WildChat
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
NeurIPS 2024 · First arXiv submission
Mines naturally occurring jailbreak tactics and combines them into adversarial tests and synthetic safety-training examples. The resulting WildJailbreak resource contains harmful and benign requests in both direct and adversarial forms, enabling experiments on safety and over-refusal.
Data used
Single-turn WildChat and LMSYS-Chat-1M examples filtered for potentially harmful interactions; WildChat contributes 8,981 adversarial queries to the tactic inventory.
Findings
WildChat and LMSYS-Chat-1M yield 5,688 tactic clusters. The synthetic WildJailbreak dataset contains approximately 262,000 prompt-response pairs.
Section 2 and Table 1: WildChat and LMSYS contributions. Sections 4–5: WildJailbreak and safety-training experiments. Read the source.
Dates and versions
First arXiv version. ArXiv version history.
NeurIPS 2024 proceedings paper.
Cite this paper
Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, Nouha Dziri. 2024. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models. Advances in Neural Information Processing Systems.
Show BibTeX
@inproceedings{wildteaming2024,
title = {{WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models}},
author = {Jiang, Liwei and Rao, Kavel and Han, Seungju and Ettinger, Allyson and Brahman, Faeze and Kumar, Sachin and Mireshghallah, Niloofar and Lu, Ximing and Sap, Maarten and Choi, Yejin and Dziri, Nouha},
year = {2024},
booktitle = {Advances in Neural Information Processing Systems},
volume = {37},
doi = {10.52202/079017-1493},
url = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/54024fca0cef9911be36319e622cde38-Abstract-Conference.html}
}