Verbalized-Sampling-Open-Ended-QA

This dataset demonstrates how Verbalized Sampling (VS) increases diversity in open-ended question answering while maintaining response quality. From the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity.

Dataset Description

The Open-Ended QA dataset contains diverse responses from state-of-the-art LLMs to open-ended questions across various domains. This dataset evaluates:

• Response diversity: Coverage of the answer space
• Quality maintenance: Whether diverse responses remain high-quality
• Creative exploration: Ability to generate novel yet relevant answers
• Distribution coverage: How well different methods explore possible answers

Key Findings

Verbalized Sampling methods achieve 2-3x higher diversity in response distributions while maintaining comparable quality to direct prompting. Methods like Multi-turn and Sequence are particularly effective at exploring the answer space.

Dataset Structure

Configs/Subsets

Each config represents a different prompting method:

• Direct: Standard prompting without special instructions (1 sample per question)
• CoT: Direct prompting with chain-of-thought (1 sample per question)
• Sequence: Sequential verbalized generation (20 samples per question)
• Multi-turn: Multi-turn verbalized sampling (20 samples per question)
• VS-Standard: Structured output with explicit probabilities (20 samples per question)
• VS-CoT: Verbalized Sampling with chain-of-thought reasoning (20 samples per question)
• VS-Multi: Combined verbalized sampling approach (20 samples per question)

Splits

Each split represents a different language model:

• gpt_4_1: GPT-4.1
• gpt_4_1_mini: GPT-4.1-mini
• claude_4_sonnet: Claude-4-Sonnet
• deepseek_r1: DeepSeek-R1
• gemini_2_5_pro: Gemini-2.5-Pro
• gemini_2_5_flash: Gemini-2.5-Flash
• o3: O3
• qwen3_235b: Qwen3-235B

Data Fields

Each example contains:

• prompt (string): The open-ended question (e.g., "Name a Japanese prefecture. Only provide the answer without explanation or punctuation.")
• responses (string): JSON-encoded list of response objects, each containing:
  • text (string): The generated answer
  • probability (float): The model's verbalized probability for this response
• model (string): The model identifier (matches the split name)

Resources

• 📄 Paper: https://arxiv.org/abs/2510.01171
• 🏠 Homepage: https://www.verbalized-sampling.com/
• 💻 GitHub: https://github.com/CHATS-lab/verbalize-sampling
• 📦 PyPI Package: https://pypi.org/project/verbalized-sampling/
• 📝 Blog: https://simonucl.notion.site/verbalized-sampling
• 🎨 Examples: https://tinyurl.com/vs-gallery
• 💡 Practical Tips: https://x.com/dch/status/1978471395173740900
• 🎙️ Podcast: https://www.youtube.com/watch?v=VoBdywmdim0

Usage

Loading the Dataset

from datasets import load_dataset

# Load a specific method configuration
dataset = load_dataset("CHATS-Lab/Verbalized-Sampling-Open-Ended-QA", "Multi-turn")

# Access data from a specific model
claude_data = dataset["claude_4_sonnet"]

print(f"Number of questions: {len(claude_data)}")

Analyzing Response Diversity

import json
from collections import Counter

# Calculate unique responses per question
for example in claude_data[:5]:  # First 5 questions
    responses = json.loads(example["responses"])
    answers = [r["text"] for r in responses]
    unique_answers = set(answers)

    print(f"\nPrompt: {example['prompt'][:50]}...")
    print(f"Total responses: {len(answers)}")
    print(f"Unique responses: {len(unique_answers)}")
    print(f"Diversity ratio: {len(unique_answers)/len(answers):.2%}")
    print(f"Sample answers: {list(unique_answers)[:5]}")

Probability Analysis

import json
import numpy as np

def analyze_probabilities(dataset_split):
    """Analyze probability distributions in responses"""
    all_probs = []

    for example in dataset_split:
        responses = json.loads(example["responses"])
        probs = [r["probability"] for r in responses]
        all_probs.extend(probs)

    return {
        "mean": np.mean(all_probs),
        "std": np.std(all_probs),
        "min": np.min(all_probs),
        "max": np.max(all_probs)
    }

# Compare probability distributions across methods
methods = ["Direct", "Multi-turn", "Sequence"]
for method in methods:
    data = load_dataset("CHATS-Lab/Verbalized-Sampling-Open-Ended-QA", method)["gpt_4_1"]
    stats = analyze_probabilities(data)
    print(f"\n{method}:")
    print(f"  Mean probability: {stats['mean']:.3f}")
    print(f"  Std probability: {stats['std']:.3f}")

Computing Diversity Metrics

import json
from nltk.util import ngrams
from collections import Counter

def compute_diversity(responses, n=2):
    """Compute n-gram diversity (distinct-n metric)"""
    all_ngrams = []

    for example in responses:
        parsed = json.loads(example["responses"])
        for resp in parsed:
            words = resp["text"].lower().split()
            all_ngrams.extend(list(ngrams(words, n)))

    unique_ngrams = len(set(all_ngrams))
    total_ngrams = len(all_ngrams)

    return unique_ngrams / total_ngrams if total_ngrams > 0 else 0

# Compare diversity across methods
direct = load_dataset("CHATS-Lab/Verbalized-Sampling-Open-Ended-QA", "Direct")["gpt_4_1"]
vs = load_dataset("CHATS-Lab/Verbalized-Sampling-Open-Ended-QA", "Multi-turn")["gpt_4_1"]

print(f"Direct distinct-2: {compute_diversity(direct, n=2):.3f}")
print(f"VS distinct-2: {compute_diversity(vs, n=2):.3f}")

Dataset Statistics

• Number of Questions: Varies by method (typically 20-50 questions)
• Responses per Question: 1 (direct methods) or 20 (verbalized methods)
• Number of Models: 8
• Number of Methods: 7
• Total Examples: ~5,000+ across all configurations
• Question Types: Geographic knowledge, creative prompts, factual queries, open-ended tasks

Dataset Creation

Source Data

Open-ended questions were selected to have multiple valid answers with varying popularity, allowing for evaluation of:

• Coverage of common answers (high-probability)
• Exploration of less common but valid answers (low-probability)
• Balance between diversity and correctness

Data Collection

For each (model, method) combination:

1. Models were prompted with identical open-ended questions
2. Multiple samples were collected (1-20 depending on method)
3. Responses were parsed and validated
4. Probabilities were extracted from verbalized outputs (when applicable)
5. Quality filtering ensured responses were on-topic and coherent

Annotations

No human annotations are present. All data consists of model-generated outputs. However, responses were automatically validated for:

• Format correctness
• Non-empty content
• Basic relevance to the prompt

Example Questions

The dataset includes diverse question types:

• Geographic: "Name a Japanese prefecture"
• Creative: "Suggest a unique baby name"
• Factual: "Name a programming language"
• Open-ended: "What's a good weekend activity?"
• Cultural: "Name a famous painting"

Considerations for Using the Data

Discussion of Biases

This dataset reveals and quantifies several LLM biases:

• Popularity bias: Models favor well-known answers (e.g., "Tokyo" over obscure prefectures)
• Recency bias: Preference for recent or trending items
• Western bias: Cultural skew in certain domains
• Mode collapse: Repetitive answers in direct prompting

Verbalized Sampling methods mitigate these biases by exploring the full answer distribution.

Quality Considerations

• Increased diversity may include some lower-quality responses
• Most verbalized responses maintain high relevance and coherence
• Quality vs. diversity tradeoff can be controlled via probability thresholds
• Direct comparison shows VS maintains competitive quality while improving diversity

Limitations

• English language only
• Limited to relatively short-form responses
• Models tested were state-of-the-art as of late 2024/early 2025
• Some questions may have culturally-specific or time-dependent answers
• Probability calibration varies across models

Dataset License Agreement

This Agreement contains the terms and conditions that govern your access and use of the Verbalized-Sampling-Open-Ended-QA Dataset. You may not use this dataset if you do not accept this Agreement. By accessing this dataset, you hereby agree to the terms of the Agreement.

Key Terms:

• ✅ Permitted: Research use, commercial use, model training, diversity evaluation, bias analysis
• ✅ Attribution Required: Must cite the original paper
• ⚠️ Restrictions: Must comply with original model terms of use, cannot redistribute independently outside HuggingFace
• ❌ Prohibited: Attempting to identify individuals, violating model-specific terms

Subject to your compliance with these terms, we grant you a limited, non-exclusive, non-transferable, non-sublicensable license to use the dataset for research, development, and improvement of software, algorithms, machine learning models, techniques, and technologies for both research and commercial purposes.

Limitation of Liability: IN NO EVENT WILL WE BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, PUNITIVE, SPECIAL, OR INDIRECT DAMAGES ARISING OUT OF OR RELATING TO THIS AGREEMENT.

Citation

If you use this dataset in your research, please cite:

@misc{zhang2025verbalizedsamplingmitigatemode,
  title={Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity},
  author={Jiayi Zhang and Simon Yu and Derek Chong and Anthony Sicilia and Michael R. Tomz and Christopher D. Manning and Weiyan Shi},
  year={2025},
  eprint={2510.01171},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2510.01171}
}

Contact

For questions or issues, please open an issue on our GitHub repository.