Bibliography
Research on chatbot use, safety, evaluation, and training.
Papers introducing, exploring, and using WildChat and WildVis, with summaries, findings, and citations.
Literature checked 2026-10-19 · 32 papers · Download the bibliography
Showing 32 papers
No papers match these filters. Try a broader search or reset the filters.
Introduces consented, multilingual, multi-turn conversations with GPT-3.5 and GPT-4, together with timestamps and metadata for studying use across time and locations. The paper examines language and prompt diversity, toxicity and jailbreaks, and instruction tuning with WildLlama.
WildChat covers 68 languages with more than 100 user prompts each. The paper demonstrates instruction tuning with a Llama-2 7B model.
Turns challenging user requests from WildChat into an evaluation set with task-specific checklists. The selection process combines language and quality filters, model difficulty estimates, and manual review. Selected conversations are held out of the public WildChat release.
For the top-ranked models evaluated in the paper, WB-Reward has a Pearson correlation of 0.98 with Chatbot Arena's human-voted Elo ratings.
Matches conversations in WildChat to published articles from two news agencies, documenting how source material and generated drafts enter journalistic workflows. The study examines input materials, overlap with published text, and publication timing.
The authors match 78 conversations to 111 published articles and report a median output-to-publication ROUGE-L overlap of 0.62.
Mines naturally occurring jailbreak tactics and combines them into adversarial tests and synthetic safety-training examples. The resulting WildJailbreak resource contains harmful and benign requests in both direct and adversarial forms, enabling experiments on safety and over-refusal.
WildChat and LMSYS-Chat-1M yield 5,688 tactic clusters. The synthetic WildJailbreak dataset contains approximately 262,000 prompt-response pairs.
Builds a moderation model that jointly identifies harmful requests, harmful responses, and refusals. WildChat contributes real user prompts and, through the WildTeaming pipeline, observed jailbreak tactics used to construct adversarial examples.
WildGuardMix contains roughly 92,000 examples, including a 5,299-item human-annotated test set. The resource supports both safety evaluation and moderation-model training.
Provides visual analysis at the group, conversation, and individual-turn levels to find jailbreak attempts in conversation logs. Analysts combine filtering, cluster views, conversation timelines, and comparisons with previously reported prompts; a case study uses WildChat to investigate multi-turn attempts after an initial model refusal.
The WildChat case study filters for multi-turn interactions and moderation scores to narrow the search to 503 conversations. Evaluation includes two case studies and interviews with nine researchers.
Studies the tasks and contexts in which people disclose personal or sensitive information to chatbots. It combines automated PII detection with task and sensitive-topic taxonomies, releasing annotations that support analyses beyond conventional identifiers.
Sensitive disclosures in the analyzed sample include preferences, personal habits, and application-related information that conventional PII detectors miss.
Converts naturally occurring user feedback into preferred and dispreferred responses. Its pipeline detects satisfaction and dissatisfaction, constructs preference examples, and evaluates responses against checklists derived from the feedback.
WildFeedback contains 20,281 preference examples. Fine-tuned models improve on the paper’s alignment and checklist-based evaluations.
Provides search filters and an embedding map for exploring WildChat and LMSYS-Chat-1M. Its case studies connect individual conversations to broader topic patterns and reproduce examples from research on journalism and personal disclosures.
The visual comparison identifies a Midjourney prompt-generation cluster in WildChat and a chemistry cluster in LMSYS-Chat-1M.
Uses WildChat prompts to compare responses from more than 50 open-weight model checkpoints and study the choice of synthetic-data generator. The authors also construct Re-Wild, a supervised fine-tuning mixture, and test it with Llama-3.1 8B.
Re-Wild outperforms the Tulu-3 SFT mixture on the paper’s aggregate evaluations using 40% as many training samples.
Uses conversations found through WildVis to study safety failures during multi-step and multilingual interactions. The paper introduces HarmScore to assess the actionability and informativeness of harmful responses, and tests these interaction patterns across four safety benchmarks.
Figure 2 uses WildVis conversations to illustrate the interaction patterns tested in the safety evaluation.
Builds Southeast Asian evaluation sets from local sources and uses real conversations to test how closely they reflect everyday requests. WildChat and LMSYS-Chat-1M supply the Wild Queries comparison corpus after language checks, length limits, and filtering.
In embedding-based comparisons, the locally constructed SeaExam and SeaBench are closer to the observed query distribution than translated MMLU and MT-Bench.
Uses WildVis to check that prompt variations in a hallucination study resemble real human requests. The paper studies cases where models give confident incorrect answers after small prompt changes despite consistently answering the original question correctly.
The authors use WildVis searches and WildChat examples to validate the help-seeking language used in their evaluation prompts.
Analyzes how users and assistants establish mutual understanding, alongside Bing Chat and human-human MultiWOZ conversations. It releases grounding annotations, a forecasting approach, and RIFTS to test when assistants should clarify, ask follow-up questions, or proceed.
RIFTS contains 1,740 tasks. The evaluated assistants rarely take useful grounding initiative; forecasting-based prompting improves their performance.
Compares topics, conversation length, sentiment, moderation scores, and inferred satisfaction across 28 languages using WildChat and LMSYS-Chat-1M. The study combines corpus analysis with a classifier for user–assistant–user interaction triples.
Topics substantially overlap across language groups, while conversation length and moderation scores vary. A model estimates satisfaction from user–assistant–user interaction triples.
Studies how users steer writing across successive turns. The authors label follow-up types, identify recurring collaboration patterns with principal component analysis, and relate those patterns to writing intents such as brainstorming, technical drafting, and long-form composition.
Seven collaboration patterns account for 80–85% of the variance in the analyzed follow-up representations; explicit positive or negative feedback appears in only a small minority of sessions.
Uses keyword searches and manual review in WildVis to select realistic goals for text rewriting. The study tests whether models can change one property, such as reading difficulty or formality, while controlling other properties.
WildVis examples support the choice of rewriting goals. The experiments find miscalibration and unintended changes to other text properties.
Introduces aggregative question answering: answering questions that require combining evidence across many conversations. WildChat-AQA pairs conversation text and metadata with inferred topics and keywords, and compares retrieval, fine-tuning, and the PROBE retrieval method.
The benchmark contains 6,027 questions over 182,330 conversations. PROBE improves over standard retrieval, with retrieval and reasoning errors remaining in the evaluated systems.
Builds a reward-model benchmark around previously unseen human prompts from WildChat, combining quality and topic filtering with manual verification. It evaluates reward models across factuality, precise instruction following, math, safety, focus, and ties, and studies how benchmark scores relate to best-of-N sampling and RLHF.
WildChat supplies the human-prompt domains: factuality (475), precise instruction following (160), math (183), and focus (495). The safety and ties domains use other prompt sources.
Examines the frequency, meaning, and usefulness of implicit feedback in real conversations. It separates feedback polarity from what users actually request, and tests whether regenerating responses with that additional context improves model training.
Feedback semantics help on MTBench but do not consistently improve the longer, more complex WildBench tasks; positive feedback alone can be a noisy training signal.
Introduces ReCCRE, which separates request content, context, assigned roles, and general expressions in user prompts. The study tracks how users phrase requests as they gain experience and over calendar time.
Users include more context in their requests over time, and individual users’ expression patterns tend to stabilize with experience.
Examines factual information conveyed by chatbot responses, including when the user's main task is editing or creative writing. It releases extracted claims and check-worthiness annotations to study information access beyond explicit questions.
WildClaims contains 121,905 extracted claims from 7,587 assistant utterances; estimated check-worthy-conversation prevalence varies substantially with the annotation method.
Examines the security and quality of code generated in real user interactions, together with user intent and follow-up behavior. The extended study also tests security-oriented prompting strategies using prompts derived from the observed logs.
Security concerns are uncommon in the analyzed user requests, even when generated code has detected problems.
Trains reward models directly on implicit user feedback using ordinal regression. Its pipeline classifies feedback, filters ambiguous signals, and validates refusal cases before training; experiments examine calibration, cross-prompt score consistency, user diversity, and online DPO.
Broader user diversity improves reward modeling. WildReward models also improve online DPO results over the reward-model baseline tested in the paper.
Develops a taxonomy of eight failure patterns that may occur without an explicit user complaint. The authors use model-assisted annotations with human oversight to analyze failures and compare original responses with newer models on a sampled set of prompts.
Among conversations labeled as failures in the annotated sample, 79% were invisible: the user gave no overt signal of the problem.
Relates inferred AI fluency to task complexity and conversational outcomes. The study compares active iteration, critical evaluation, and passive acceptance, and examines how these behaviors relate to visible and invisible failures.
More fluent users attempt harder tasks and show more visible failures, yet achieve greater success on complex work; less fluent users more often have failures without overt negative feedback.
Tests whether learning a user simulator from real conversations helps train assistants that work better with people. The study fine-tunes simulators on WildChat user utterances, trains assistants through simulated interaction, and evaluates them on WildBench and in a study with 283 participants.
The assistant trained with the learned user simulator wins 58% of comparisons against the initial assistant in the human study; training with a role-playing model yields 51%.
Compares individual and population-level usage patterns in Bing Copilot and WildChat-4.8M. The study tracks changes over a user’s history and compares users with different activity levels across the two platforms.
WildChat users’ intent and domain distributions more closely resemble high-activity Copilot users than low-activity users.
Studies how people generate and consume fiction through chatbots, combining conversation classification with close reading of repeated story requests. The authors examine fanfiction, roleplay, and erotic fiction, and discuss how iterative generation can reshape the relationship between reader and writer. A companion viewer supports exploration of the fiction subset.
The study classifies 195,271 conversations (34%) as fiction. Sampling one conversation per inferred user reduces this share to 7.1%.
Examines user-side politeness in Arabic conversations selected through WildVis. The authors code prompts with an adapted Brown and Levinson framework and compare openings, conversation bodies, closings, and topic shifts to study directness and relational language in human–AI interaction.
Bald on-record strategies account for 88.42% of coded instances in the selected conversations.
Studies how the definition of a mental-health conversation changes prevalence estimates in WildChat-4.8M. A classifier scores topicality, help-seeking intent, clinical language, and affective risk, with validation against independently coded conversations.
The estimated share of mental-health conversations is 0.21% when the definition requires explicit help-seeking for an identifiable person, and 4.90% when any mental-health topicality qualifies.
Introduces WildStories, containing 275,635 story prompts, and WildEdits, containing 24,291 reconstructed edit trees. The study follows revisions across separate conversations, organizes edits by direction and narrative target, and uses this framework to generate story-prompt variations for evaluation.
94.2% of prompts with an identified parent link to a different WildChat conversation, showing how story revisions span separate conversations.
Aaron Snoswell’s advanced-search fork of WildVisualizer adds compound phrase searches, excluded phrases, and user/assistant turn filters.