Verbalized-Sampling: Random-Number-Generator

This dataset evaluates the effectiveness of Verbalized Sampling (VS) in generating uniform random distributions, a task where LLMs typically exhibit significant mode collapse. From the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity.

Dataset Description

The Random Number Generator (RNG) dataset contains responses from various state-of-the-art LLMs asked to perform simple random generation tasks, such as rolling a fair six-sided dice. This dataset is crucial for evaluating:

• Uniformity of distribution: How evenly spread are the generated values?
• Mode collapse: Do models repeatedly favor certain outcomes?
• Method effectiveness: How well do different prompting strategies mitigate bias?

Dataset Structure

Configs/Subsets

Each config represents a different prompting method:

• Direct: Standard prompting without special instructions (1 sample)
• CoT: Direct prompting with chain-of-thought (1 sample)
• Sequence: Sequential verbalized generation (5 samples)
• Multi-turn: Multi-turn verbalized sampling (5 samples)
• VS-Standard: Structured output with explicit probabilities (5 samples)
• VS-CoT: Verbalized Sampling with chain-of-thought reasoning (5 samples)
• VS-Multi: Combined verbalized sampling approach (5 samples)

Splits

Each split represents a different language model:

• gpt_4_1: GPT-4.1
• gpt_4_1_mini: GPT-4.1-mini
• claude_4_sonnet: Claude-4-Sonnet
• deepseek_r1: DeepSeek-R1
• gemini_2_5_pro: Gemini-2.5-Pro
• gemini_2_5_flash: Gemini-2.5-Flash
• o3: o3 from OpenAI
• qwen3-235b: Qwen3 235B A22B Instruct 2507

Data Fields

Each example contains:

• prompt (string): The random generation task (e.g., "Roll a fair six-sided dice. Return ONLY the integer result (1-6), with no explanation or extra text.")
• responses (string): JSON-encoded list of response objects, each containing:
  • text (string): The generated response
  • probability (float): The model's verbalized probability for this response
• model (string): The model identifier (matches the split name)

Resources

• 📄 Paper: https://arxiv.org/abs/2510.01171
• 🏠 Homepage: https://www.verbalized-sampling.com/
• 💻 GitHub: https://github.com/CHATS-lab/verbalize-sampling
• 📦 PyPI Package: https://pypi.org/project/verbalized-sampling/
• 📝 Blog: https://simonucl.notion.site/verbalized-sampling
• 🎨 Examples: https://tinyurl.com/vs-gallery
• 🎙️ Podcast: https://www.youtube.com/watch?v=VoBdywmdim0

Usage

Loading the Dataset

from datasets import load_dataset

# Load a specific method configuration
dataset = load_dataset("CHATS-Lab/Verbalized-Sampling-Random-Number-Generator", "Multi-turn")

# Access data from a specific model
gpt_data = dataset["gpt_4_1"]

print(f"Number of examples: {len(gpt_data)}")

Parsing Responses

import json
from collections import Counter

# Analyze distribution of dice rolls
all_rolls = []
for example in gpt_data:
    responses = json.loads(example["responses"])
    for resp in responses:
        all_rolls.append(int(resp["text"]))

# Calculate distribution
distribution = Counter(all_rolls)
print("Distribution:", dict(sorted(distribution.items())))

# Calculate uniformity (ideal is 1/6 ≈ 0.167 for each outcome)
import numpy as np
values = [distribution.get(i, 0) for i in range(1, 7)]
uniformity = np.std(values) / np.mean(values)  # Lower is more uniform
print(f"Coefficient of variation: {uniformity:.3f}")

Comparing Methods

from datasets import load_dataset
import json
import numpy as np

def calculate_entropy(responses):
    """Calculate Shannon entropy of the distribution"""
    from collections import Counter
    import math

    values = []
    for example in responses:
        parsed = json.loads(example["responses"])
        values.extend([int(r["text"]) for r in parsed])

    counts = Counter(values)
    total = sum(counts.values())
    probs = [c/total for c in counts.values()]

    return -sum(p * math.log2(p) for p in probs if p > 0)

# Compare Direct vs Multi-turn
direct_data = load_dataset("CHATS-Lab/Verbalized-Sampling-Random-Number-Generator", "Direct")["gpt_4_1"]
multiturn_data = load_dataset("CHATS-Lab/Verbalized-Sampling-Random-Number-Generator", "Multi-turn")["gpt_4_1"]

print(f"Direct entropy: {calculate_entropy(direct_data):.3f}")
print(f"Multi-turn entropy: {calculate_entropy(multiturn_data):.3f}")
print(f"Ideal entropy (uniform): {np.log2(6):.3f}")

Dataset Statistics

• Total Examples per Method: ~120 examples (varies by method and model)
• Number of Models: 7
• Number of Methods: 7
• Total Examples: ~840 per method configuration
• Task Types: Dice rolling, number generation, uniform sampling

Dataset Creation

Source Data

Random generation tasks were carefully designed to have known uniform distributions, allowing for objective evaluation of model behavior.

Data Collection

For each (model, method) combination:

1. Models were prompted with identical random generation tasks
2. Multiple samples were collected per prompt (1-5 depending on method)
3. Responses were parsed and validated for correctness
4. Probabilities were extracted from verbalized outputs (when applicable)

Annotations

No human annotations are present in this dataset. All data consists of model-generated outputs.

Considerations for Using the Data

Discussion of Biases

This dataset reveals inherent biases in LLMs:

• Digit preference: Models often prefer certain numbers (e.g., 3, 7) over others
• Range bias: Tendency to avoid extreme values
• Pattern following: Models may generate non-random patterns

These biases are intentionally preserved in the dataset to demonstrate the effectiveness of Verbalized Sampling in mitigating them.

Limitations

• Limited to simple random generation tasks (dice rolls, number generation)
• Only evaluates uniformity, not other aspects of randomness (e.g., sequence unpredictability)
• Models tested were state-of-the-art as of late 2024/early 2025
• English language only

Dataset License Agreement

This Agreement contains the terms and conditions that govern your access and use of the Verbalized-Sampling-Random-Number-Generator Dataset. You may not use this dataset if you do not accept this Agreement. By accessing this dataset, you hereby agree to the terms of the Agreement.

Key Terms:

• ✅ Permitted: Research use, commercial use, model training, algorithm development
• ✅ Attribution Required: Must cite the original paper
• ⚠️ Restrictions: Must comply with original model terms of use, cannot redistribute independently outside HuggingFace
• ❌ Prohibited: Attempting to identify individuals, violating model-specific terms

Subject to your compliance with these terms, we grant you a limited, non-exclusive, non-transferable, non-sublicensable license to use the dataset for research, development, and improvement of software, algorithms, machine learning models, techniques, and technologies for both research and commercial purposes.

Limitation of Liability: IN NO EVENT WILL WE BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, PUNITIVE, SPECIAL, OR INDIRECT DAMAGES ARISING OUT OF OR RELATING TO THIS AGREEMENT.

Citation

If you use this dataset in your research, please cite:

@misc{zhang2025verbalizedsamplingmitigatemode,
  title={Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity},
  author={Jiayi Zhang and Simon Yu and Derek Chong and Anthony Sicilia and Michael R. Tomz and Christopher D. Manning and Weiyan Shi},
  year={2025},
  eprint={2510.01171},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2510.01171}
}

Contact

For questions or issues, please open an issue on our GitHub repository.