Verbalized-Sampling-Dialogue-Simulation

This dataset demonstrates how Verbalized Sampling (VS) enables more diverse and realistic multi-turn conversational simulations between AI agents. From the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity.

Dataset Description

The Dialogue Simulation dataset contains multi-turn conversations between pairs of language models, comparing different approaches to generating diverse social interactions. This dataset is valuable for:

• Social simulation: Modeling realistic human-like conversations
• Conversational diversity: Exploring different dialogue trajectories
• Multi-agent interaction: Understanding how LLMs communicate with each other
• Behavioral research: Studying emergent conversational patterns

Key Findings

Verbalized Sampling approaches produce significantly more diverse dialogue trajectories compared to baseline methods, while maintaining conversational coherence and naturalness. This enables more realistic social simulations and better coverage of possible conversation paths.

Dataset Structure

Configs/Subsets

Each config represents a different dialogue generation method:

• Direct: Standard prompting baseline without special sampling
• Fine_Tune: Conversations using fine-tuned models
• VS_Standard: Verbalized Sampling with structured probabilities
• VS_CoT: Verbalized Sampling with chain-of-thought (if available)
• VS_Multi: Combined verbalized sampling approach (if available)

Splits

Each split represents a specific model used in the conversations:

Available models:

• gpt_4_1: GPT-4.1
• gpt_4_1_mini: GPT-4.1-mini
• claude_4_sonnet: Claude-4-Sonnet
• gemini_2_5_pro: Gemini-2.5-Pro
• gemini_2_5_flash: Gemini-2.5-Flash
• deepseek_r1: DeepSeek-R1
• o3: o3 from OpenAI
• Qwen3-235b: Qwen3 235B A22B Instruct 2507
• llama_3_1_70b_instruct: Llama-3.1-70B-Instruct
• Llama_3_1_8B: Llama-3.1-8B (Fine-Tuned) - only available in Fine_Tune config

Data Fields

Each example contains:

• conversation_id (string): Unique identifier for the conversation
• utterances (string): JSON-encoded list of conversation turns, each containing:
  • conversation_id (string): Links utterance to conversation
  • utterance_id (string): Unique ID for this utterance
  • speaker (string): Speaker identifier
  • role (int): Speaker role (0 or 1)
  • text (string): The utterance content
• model (string): The model used for this conversation
• method (string): The generation method (Direct, Fine_Tune, VS_Standard, etc.)
• num_turns (int): Total number of utterances in the conversation

Resources

• 📄 Paper: https://arxiv.org/abs/2510.01171
• 🏠 Homepage: https://www.verbalized-sampling.com/
• 💻 GitHub: https://github.com/CHATS-lab/verbalize-sampling
• 📦 PyPI Package: https://pypi.org/project/verbalized-sampling/
• 📝 Blog: https://simonucl.notion.site/verbalized-sampling
• 🎨 Examples: https://tinyurl.com/vs-gallery
• 💡 Practical Tips: https://x.com/dch/status/1978471395173740900
• 🎙️ Podcast: https://www.youtube.com/watch?v=VoBdywmdim0

Usage

Loading the Dataset

from datasets import load_dataset

# Load Direct method conversations
direct = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation", "Direct")

# Access GPT-4.1 conversations
gpt_convos = direct["gpt_4_1"]

print(f"Number of conversations: {len(gpt_convos)}")

Parsing Conversations

import json

# Read and display a conversation
for example in gpt_convos[:1]:  # First conversation
    print(f"Conversation ID: {example['conversation_id']}")
    print(f"Model: {example['model']}")
    print(f"Method: {example['method']}")
    print(f"Number of turns: {example['num_turns']}\n")

    utterances = json.loads(example["utterances"])

    # Display conversation
    for utt in utterances:
        speaker_label = "Speaker A" if utt["role"] == 0 else "Speaker B"
        print(f"{speaker_label}: {utt['text']}\n")

Analyzing Conversation Length

import json
import numpy as np

def analyze_conversation_lengths(dataset_split):
    """Compute statistics about conversation lengths"""
    lengths = []

    for example in dataset_split:
        utterances = json.loads(example["utterances"])
        lengths.append(len(utterances))

    return {
        "mean": np.mean(lengths),
        "median": np.median(lengths),
        "min": np.min(lengths),
        "max": np.max(lengths),
        "std": np.std(lengths)
    }

# Compare conversation lengths across methods
direct_data = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation", "Direct")["gpt_4_1"]
vs_standard_data = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation, "VS_Standard")["gpt_4_1"]

print("Direct:", analyze_conversation_lengths(direct_data))
print("VS_Standard:", analyze_conversation_lengths(vs_standard_data))

Computing Diversity Metrics

import json
from collections import Counter

def compute_lexical_diversity(dataset_split):
    """Compute lexical diversity (type-token ratio)"""
    all_words = []

    for example in dataset_split:
        utterances = json.loads(example["utterances"])
        for utt in utterances:
            words = utt["text"].lower().split()
            all_words.extend(words)

    unique_words = len(set(all_words))
    total_words = len(all_words)

    return unique_words / total_words if total_words > 0 else 0

def compute_topic_diversity(dataset_split):
    """Measure diversity of conversation topics"""
    # Simple topic extraction using first few words
    topics = []

    for example in dataset_split:
        utterances = json.loads(example["utterances"])
        if utterances:
            # Use first utterance as topic indicator
            first_utt = utterances[0]["text"][:50]
            topics.append(first_utt)

    unique_topics = len(set(topics))
    total_convos = len(topics)

    return unique_topics / total_convos if total_convos > 0 else 0

direct = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation", "Direct")["gpt_4_1"]
vs_standard = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation", "VS_Standard")["gpt_4_1"]

print(f"Direct lexical diversity: {compute_lexical_diversity(direct):.3f}")
print(f"VS_Standard lexical diversity: {compute_lexical_diversity(vs_standard):.3f}")
print(f"Direct topic diversity: {compute_topic_diversity(direct):.3f}")
print(f"VS_Standard topic diversity: {compute_topic_diversity(vs_standard):.3f}")

Turn-Taking Analysis

import json

def analyze_turn_taking(dataset_split):
    """Analyze turn-taking patterns"""
    turn_counts = {0: 0, 1: 0}
    total_switches = 0

    for example in dataset_split:
        utterances = json.loads(example["utterances"])

        for utt in utterances:
            turn_counts[utt["role"]] += 1

        # Count role switches
        for i in range(1, len(utterances)):
            if utterances[i]["role"] != utterances[i-1]["role"]:
                total_switches += 1

    total_turns = sum(turn_counts.values())
    balance = min(turn_counts.values()) / max(turn_counts.values()) if max(turn_counts.values()) > 0 else 0

    return {
        "total_turns": total_turns,
        "speaker_balance": balance,
        "avg_switches_per_convo": total_switches / len(dataset_split)
    }

# Analyze turn-taking patterns
direct = load_dataset("CHATS-Lab/Verbalized-Sampling-Dialogue-Simulation", "Direct")["gpt_4_1"]
print("Turn-taking analysis:", analyze_turn_taking(direct))

Dataset Statistics

• Total Conversations: Varies by method and model (typically 50-200 per model)
• Number of Models: 7 (6 for most methods, 1 for Fine_Tune)
• Number of Methods: 5 (Direct, Fine_Tune, VS_Standard, VS_CoT, VS_Multi)
• Average Conversation Length: 10-20 turns
• Total Utterances: 10,000+ across all configurations

Per-Method Availability

• Direct: GPT-4.1, Claude-4-Sonnet, Gemini-2.5-Pro, Gemini-2.5-Flash, DeepSeek-R1
• Fine_Tune: Llama-3.1-8B only
• VS_Standard: GPT-4.1 (and potentially others)
• VS_CoT: Varies (check availability)
• VS_Multi: Varies (check availability)

Dataset Creation

Source Data

Conversations were generated through multi-turn interactions between pairs of language models. Initial prompts were designed to elicit natural, engaging dialogues.

Data Collection

For each (model_pair, approach) combination:

1. Initial conversational prompt provided to first model
2. Models alternated generating responses
3. Conversations continued for multiple turns (typically 10-20)
4. Verbalized Sampling applied in sampling approach to select diverse responses
5. Conversations logged with full turn-by-turn history

Data Format

Conversations are stored as JSONL with one utterance per line, grouped by conversation ID. Each utterance maintains:

• Conversation context linkage
• Speaker identity and role
• Temporal ordering (via utterance_id)

Use Cases

This dataset is valuable for:

1. Social Simulation Research: Understanding emergent social behavior in LLM interactions
2. Conversational AI Development: Training and evaluating dialogue systems
3. Diversity Evaluation: Measuring conversation diversity across different sampling methods
4. Multi-Agent Systems: Studying coordination and communication between AI agents

Considerations for Using the Data

Discussion of Biases

• Model-specific biases: Each model brings its own conversational style and biases
• Topic bias: Conversations may cluster around certain topics depending on initial prompts
• Politeness bias: Models tend toward overly polite or agreeable interactions
• Cultural bias: Reflects cultural assumptions embedded in training data

Safety and Content Notes

• This dataset contains conversations between AI models, not real humans
• Conversations are generally appropriate but may occasionally include:
  • Factual errors or hallucinations
  • Stereotypical responses
  • Repetitive patterns (especially in baseline)
• Not recommended for training conversational agents without additional filtering

Limitations

• English language only
• Simulated conversations, not human dialogues
• Limited to text-based interaction
• Models from late 2024/early 2025
• No grounding in external knowledge or events
• Lacks emotional depth of human conversation

Dataset License Agreement

This Agreement contains the terms and conditions that govern your access and use of the Verbalized-Sampling-Dialogue-Simulation Dataset. You may not use this dataset if you do not accept this Agreement. By accessing this dataset, you hereby agree to the terms of the Agreement.

Key Terms:

• ✅ Permitted: Research use, commercial use, dialogue system development, social simulation
• ✅ Attribution Required: Must cite the original paper
• ⚠️ Restrictions: Must comply with original model terms of use, cannot redistribute independently outside HuggingFace
• ❌ Prohibited: Attempting to identify individuals, claiming conversations are human-generated, violating model-specific terms

Subject to your compliance with these terms, we grant you a limited, non-exclusive, non-transferable, non-sublicensable license to use the dataset for research, development, and improvement of software, algorithms, machine learning models, techniques, and technologies for both research and commercial purposes.

Limitation of Liability: IN NO EVENT WILL WE BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, PUNITIVE, SPECIAL, OR INDIRECT DAMAGES ARISING OUT OF OR RELATING TO THIS AGREEMENT.

Citation

If you use this dataset in your research, please cite:

@misc{zhang2025verbalizedsamplingmitigatemode,
  title={Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity},
  author={Jiayi Zhang and Simon Yu and Derek Chong and Anthony Sicilia and Michael R. Tomz and Christopher D. Manning and Weiyan Shi},
  year={2025},
  eprint={2510.01171},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2510.01171}
}

Contact

For questions or issues, please open an issue on our GitHub repository.