Uses WildChat
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
NeurIPS 2024 Datasets and Benchmarks · First arXiv submission
Builds a moderation model that jointly identifies harmful requests, harmful responses, and refusals. WildChat contributes real user prompts and, through the WildTeaming pipeline, observed jailbreak tactics used to construct adversarial examples.
Data used
WildChat prompts form one component of a mixed-source moderation dataset; additional sources include LMSYS, annotator-written safety data, and synthetic adversarial examples.
Findings
WildGuardMix contains roughly 92,000 examples, including a 5,299-item human-annotated test set. The resource supports both safety evaluation and moderation-model training.
Section 3.1: WildChat sampling and tactic-based generation. Section 3.2: test annotation. Read the source.
Dates and versions
First arXiv version. ArXiv version history.
NeurIPS 2024 proceedings paper.
Cite this paper
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri. 2024. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. Advances in Neural Information Processing Systems.
Show BibTeX
@inproceedings{wildguard2024,
title = {{WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs}},
author = {Han, Seungju and Rao, Kavel and Ettinger, Allyson and Jiang, Liwei and Lin, Bill Yuchen and Lambert, Nathan and Choi, Yejin and Dziri, Nouha},
year = {2024},
booktitle = {Advances in Neural Information Processing Systems},
volume = {37},
doi = {10.52202/079017-0261},
url = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/0f69b4b96a46f284b726fbd70f74fb3b-Abstract.html}
}