Verbalized-Sampling-Joke-Generation

This dataset demonstrates how Verbalized Sampling (VS) increases diversity in creative generation tasks, specifically joke generation, while maintaining humor quality. From the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity.

Dataset Description

The Joke Generation dataset contains diverse jokes from state-of-the-art LLMs in response to prompts requesting jokes about specific topics. This dataset evaluates:

• Creative diversity: Coverage of different joke styles, structures, and punchlines
• Quality maintenance: Whether diverse jokes remain humorous and coherent
• Mode collapse mitigation: Avoiding repetitive joke patterns and structures
• Distribution coverage: How well different methods explore the creative space

Key Findings

Verbalized Sampling methods achieve 2-3x higher diversity in joke generation while maintaining comparable humor quality to direct prompting. Methods like Multi-turn and Sequence are particularly effective at exploring different comedic approaches and avoiding repetitive punchlines.

Dataset Structure

Configs/Subsets

Each config represents a different prompting method:

• Direct: Standard prompting without special instructions (1 sample per prompt)
• CoT: Direct prompting with chain-of-thought (1 sample per prompt)
• Sequence: Sequential verbalized generation (5 samples per prompt)
• Multi-turn: Multi-turn verbalized sampling (5 samples per prompt)
• VS-Standard: Structured output with explicit probabilities (5 samples per prompt)
• VS-CoT: Verbalized Sampling with chain-of-thought reasoning (5 samples per prompt)
• VS-Multi: Combined verbalized sampling approach (5 samples per prompt)

Splits

Each split represents a different language model:

• gpt_4_1: GPT-4.1
• gpt_4_1_mini: GPT-4.1-mini
• claude_3_7_sonnet: Claude-3.7-Sonnet
• claude_4_sonnet: Claude-4-Sonnet
• deepseek_r1: DeepSeek-R1
• gemini_2_5_pro: Gemini-2.5-Pro
• gemini_2_5_flash: Gemini-2.5-Flash
• o3: O3
• qwen3_235b: Qwen3-235B
• llama_3_1_70b_instruct: Llama-3.1-70B-Instruct

Data Fields

Each example contains:

• prompt (string): The joke request (e.g., "Tell me a joke about drinking")
• responses (string): JSON-encoded list of response objects, each containing:
  • text (string): The generated joke
• model (string): The model identifier (matches the split name)

Resources

• 📄 Paper: https://arxiv.org/abs/2510.01171
• 🏠 Homepage: https://www.verbalized-sampling.com/
• 💻 GitHub: https://github.com/CHATS-lab/verbalize-sampling
• 📦 PyPI Package: https://pypi.org/project/verbalized-sampling/
• 📝 Blog: https://simonucl.notion.site/verbalized-sampling
• 🎨 Examples: https://tinyurl.com/vs-gallery
• 💡 Practical Tips: https://x.com/dch/status/1978471395173740900
• 🎙️ Podcast: https://www.youtube.com/watch?v=VoBdywmdim0

Usage

Loading the Dataset

from datasets import load_dataset

# Load a specific method configuration
dataset = load_dataset("CHATS-Lab/Verbalized-Sampling-Joke-Generation", "Multi-turn")

# Access data from a specific model
claude_data = dataset["claude_4_sonnet"]

print(f"Number of joke prompts: {len(claude_data)}")

Analyzing Joke Diversity

import json
from collections import Counter

# Calculate unique jokes per prompt
for example in claude_data[:3]:  # First 3 prompts
    responses = json.loads(example["responses"])
    jokes = [r["text"] for r in responses]
    unique_jokes = set(jokes)

    print(f"\nPrompt: {example['prompt']}")
    print(f"Total responses: {len(jokes)}")
    print(f"Unique jokes: {len(unique_jokes)}")
    print(f"Diversity ratio: {len(unique_jokes)/len(jokes):.2%}")
    print("\nSample jokes:")
    for joke in list(unique_jokes)[:3]:
        print(f"  - {joke}")

Comparing Methods Side-by-Side

import json
from datasets import load_dataset

def show_jokes(method_name, model="gpt_4_1", n=1):
    """Display jokes from a specific method"""
    data = load_dataset("CHATS-Lab/Verbalized-Sampling-Joke-Generation", method_name)
    example = data[model][n]

    print(f"\n{'='*60}")
    print(f"Method: {method_name} | Model: {model}")
    print(f"{'='*60}")
    print(f"Prompt: {example['prompt']}\n")

    responses = json.loads(example["responses"])
    for i, resp in enumerate(responses, 1):
        print(f"Joke {i}: {resp['text']}\n")

# Compare different methods for the same prompt
for method in ["Direct", "Multi-turn", "Sequence"]:
    show_jokes(method, model="gpt_4_1", n=0)

Computing Diversity Metrics

import json
from nltk.util import ngrams
from collections import Counter

def compute_diversity(responses, n=2):
    """Compute n-gram diversity (distinct-n metric)"""
    all_ngrams = []

    for example in responses:
        parsed = json.loads(example["responses"])
        for resp in parsed:
            words = resp["text"].lower().split()
            all_ngrams.extend(list(ngrams(words, n)))

    unique_ngrams = len(set(all_ngrams))
    total_ngrams = len(all_ngrams)

    return unique_ngrams / total_ngrams if total_ngrams > 0 else 0

# Compare diversity across methods
direct = load_dataset("CHATS-Lab/Verbalized-Sampling-Joke-Generation", "Direct")["gpt_4_1"]
vs = load_dataset("CHATS-Lab/Verbalized-Sampling-Joke-Generation", "Multi-turn")["gpt_4_1"]

print(f"Direct distinct-2: {compute_diversity(direct, n=2):.3f}")
print(f"VS distinct-2: {compute_diversity(vs, n=2):.3f}")

Analyzing Punchline Diversity

import json
from collections import Counter

def extract_punchlines(dataset_split):
    """Extract the last sentence of each joke (likely the punchline)"""
    punchlines = []

    for example in dataset_split:
        responses = json.loads(example["responses"])
        for resp in responses:
            joke = resp["text"]
            # Simple heuristic: last sentence is the punchline
            sentences = joke.split('.')
            if len(sentences) > 1:
                punchline = sentences[-1].strip()
                if punchline:
                    punchlines.append(punchline)

    return punchlines

def analyze_punchline_diversity(method_name, model="gpt_4_1"):
    """Compare punchline repetition rates"""
    data = load_dataset("CHATS-Lab/Verbalized-Sampling-Joke-Generation", method_name)[model]
    punchlines = extract_punchlines(data)

    total = len(punchlines)
    unique = len(set(punchlines))

    print(f"\n{method_name}:")
    print(f"  Total punchlines: {total}")
    print(f"  Unique punchlines: {unique}")
    print(f"  Diversity: {unique/total:.2%}")

    # Show most repeated punchlines
    counter = Counter(punchlines)
    most_common = counter.most_common(3)
    print(f"  Most repeated:")
    for punchline, count in most_common:
        if count > 1:
            print(f"    '{punchline[:50]}...' ({count}x)")

# Compare punchline diversity across methods
for method in ["Direct", "Multi-turn", "Sequence", "VS-Standard"]:
    analyze_punchline_diversity(method)

Dataset Statistics

• Number of Prompts: Varies by method (typically 20-50 prompts)
• Responses per Prompt: 1 (direct methods) or 5 (verbalized methods)
• Number of Models: 10
• Number of Methods: 7
• Total Examples: ~3,000+ across all configurations
• Prompt Types: Topic-based jokes (e.g., "Tell me a joke about [topic]")

Dataset Creation

Source Data

Joke prompts were selected to cover diverse topics while being open-ended enough to allow for multiple creative approaches:

• Common topics: drinking, animals, technology, work, relationships
• Abstract concepts: time, decisions, knowledge
• Specific subjects: programming, science, daily activities

Data Collection

For each (model, method) combination:

1. Models were prompted with identical joke requests
2. Multiple samples were collected (1-5 depending on method)
3. Responses were parsed and validated for completeness
4. Quality filtering ensured jokes were coherent and on-topic
5. Duplicate filtering removed exact repetitions within each sample set

Annotations

No human annotations are present. All data consists of model-generated outputs. However, responses were automatically validated for:

• Format correctness
• Non-empty content
• Basic relevance to the prompt topic
• Coherence (complete sentences)

Example Prompts

The dataset includes jokes about various topics:

• Common themes: "Tell me a joke about drinking"
• Technology: "Tell me a joke about programming"
• Animals: "Tell me a joke about cats"
• Work: "Tell me a joke about meetings"
• Science: "Tell me a joke about physics"
• Abstract: "Tell me a joke about time"

Considerations for Using the Data

Discussion of Biases

This dataset reveals and quantifies several LLM biases in creative generation:

• Template bias: Models tend to reuse similar joke structures (e.g., "Why did X...?" format)
• Punchline recycling: Direct prompting often produces similar punchlines across different topics
• Safety bias: Models may avoid edgy or controversial humor
• Cultural bias: Jokes reflect Western humor conventions and cultural references
• Mode collapse: Repetitive patterns in setup-punchline structure

Verbalized Sampling methods mitigate these biases by encouraging exploration of different joke styles and structures.

Quality Considerations

• Increased diversity may include some jokes with less obvious humor
• Most verbalized responses maintain coherent structure and topic relevance
• Quality vs. diversity tradeoff reflects natural variation in humor
• Joke quality is subjective and culturally dependent
• Direct comparison shows VS maintains competitive quality while improving diversity

Limitations

• English language only
• Limited to short-form, single-joke responses
• Models tested were state-of-the-art as of late 2024/early 2025
• Humor is subjective and culturally dependent
• No human ratings of joke quality are included
• Some models may be more conservative in humor generation due to safety training

Dataset License Agreement

This Agreement contains the terms and conditions that govern your access and use of the Verbalized-Sampling-Joke-Generation Dataset. You may not use this dataset if you do not accept this Agreement. By accessing this dataset, you hereby agree to the terms of the Agreement.

Key Terms:

• ✅ Permitted: Research use, commercial use, model training, creativity evaluation, diversity analysis
• ✅ Attribution Required: Must cite the original paper
• ⚠️ Restrictions: Must comply with original model terms of use, cannot redistribute independently outside HuggingFace
• ❌ Prohibited: Attempting to identify individuals, violating model-specific terms

Subject to your compliance with these terms, we grant you a limited, non-exclusive, non-transferable, non-sublicensable license to use the dataset for research, development, and improvement of software, algorithms, machine learning models, techniques, and technologies for both research and commercial purposes.

Limitation of Liability: IN NO EVENT WILL WE BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, PUNITIVE, SPECIAL, OR INDIRECT DAMAGES ARISING OUT OF OR RELATING TO THIS AGREEMENT.

Citation

If you use this dataset in your research, please cite:

@misc{zhang2025verbalizedsamplingmitigatemode,
  title={Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity},
  author={Jiayi Zhang and Simon Yu and Derek Chong and Anthony Sicilia and Michael R. Tomz and Christopher D. Manning and Weiyan Shi},
  year={2025},
  eprint={2510.01171},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2510.01171}
}

Contact

For questions or issues, please open an issue on our GitHub repository.