Contextual Diversity Measure (CDM)

This repository contains the code and dataset for the paper:

Contextual Diversity Measure (CDM) for Controllable Story Generation in Large Language Models

CDM is a metric that quantifies semantic diversity for scenario generation given abstract semantic role labeling constraints. It introduces a geometric decomposition that analyses word-level semantic changes relative to the overall sentence-level shift, producing scores that can both evaluate generation quality and be integrated into model training objectives.

---

Overview

Repository Structure

.
├── code/
│   ├── pipeline.py              # Full pipeline: CDM computation + embedding visualisation
│   ├── embedding.py             # Word embedding loader (GloVe, Word2Vec, FastText, MiniLM)
│   ├── distance.py              # CDM distance calculator with geometric decomposition
│   ├── baseline/
│   │   ├── bleu.py              # Self-BLEU baseline
│   │   ├── bert-score.py        # BERTScore baseline
│   │   ├── sentence-transformer.py  # Sentence similarity baseline
│   │   └── distinct-n.py        # Distinct-N baseline
│   └── tools/
│       ├── diff_test_nf.py      # Wilcoxon signed-rank tests & Cohen's d
│       ├── compute_stat.py      # Accuracy, sum, and mean for CDM results
│       └── compute_stat_baseline.py  # Accuracy, sum, and mean for baseline results
├── dataset/
│   ├── prompt.txt               # Prompt used to generate the LLM-generated dataset
│   ├── llm_001.json
│   ├── llm_002.json
│   └── ...                      # 218 scenario JSON files (curated + LLM-generated)
└── result/
    ├── statistical_test.csv     # Wilcoxon tests and Cohen's d across all embeddings
    └── performance/
        ├── cdm/
        │   ├── fasttext.csv
        │   ├── glove.csv
        │   ├── minilm.csv
        │   └── w2v.csv
        └── baseline/
            ├── bleu.csv
            ├── bert.csv
            ├── sensim.csv
            └── dn.csv

---

Dataset

The dataset consists of 218 instances, each stored as a JSON file containing three text realisations of the same abstract semantic role labeling constraint.

{
  "output": "Reference sentence text...",
  "alternative": [
    "High-diversity alternative text...",
    "Low-diversity alternative text..."
  ],
  "specific_words": [
    ["word1_ref", "word2_ref", "..."],
    ["word1_alt1", "word2_alt1", "..."],
    ["word1_alt2", "word2_alt2", "..."]
  ]
}

---

Core Modules

embedding.py — Handles loading and querying word embeddings across multiple backends (GloVe, Word2Vec, FastText, MiniLM), with support for single words, multi-word phrases, stop word filtering, and batch embedding preparation.

distance.py — Implements the CDM metric as described in the paper, performing a geometric decomposition of word-level semantic changes into directional and orthogonal components relative to the centroid direction between sentences.

pipeline.py — Provides an end-to-end pipeline that:

1. Loads scenario data from JSON files (single file or batch directory mode).
2. Computes CDM distances using aligned word embeddings.
3. Computes centroid distances (Euclidean and cosine).
4. Generates word-by-word distance comparison matrices.
5. Produces 3D embedding visualisations using dimensionality reduction (UMAP, t-SNE, or PCA) from multiple viewing angles, with colouring by sentence or by word position.

Results

The result/ directory contains pre-computed outputs:

• statistical_test.csv — Wilcoxon two-sided, one-sided p-values, and Cohen's d across all four embedding models on both the manually curated and LLM-generated subsets.
• performance/cdm/ — Per-scenario CDM(P1) − CDM(P2) difference values for each embedding model.
• performance/baseline/ — Per-scenario difference values for each baseline method (Self-BLEU, BERTScore, Sentence Similarity, Distinct-N).

---

Getting Started

Installation

pip install -r requirements.txt

Or install manually:

pip install torch numpy scipy pandas scikit-learn matplotlib seaborn umap-learn nltk gensim transformers sentence-transformers bert-score

Running the Full Pipeline

Process a single scenario:

python -m code/pipeline.py dataset/llm_001.json \
    -m glove \
    -o output/ \
    -r umap \
    -t 0.5 -u 1.9 -v 1.7

Process all scenarios in a directory:

python -m code/pipeline.py dataset/ \
    -m fasttext \
    -o output/ \
    -r umap \
    -t 0.5 -u 1.6 -v 1.7

Arguments:

Running Baseline Metrics

# Self-BLEU
python code/baseline/bleu.py dataset/ -o self_bleu_results.csv
 
# BERTScore
python code/baseline/bert-score.py dataset/ -o bertscore_results.csv
 
# Sentence Similarity
python code/baseline/sentence-transformer.py dataset/ -o sentence_similarity_results.csv
 
# Distinct-N
python code/baseline/distinct-n.py dataset/ -o distinct_n_results.csv

Statistical Testing

Run Wilcoxon signed-rank tests and compute Cohen's d on CDM results:

# Single file
python code/tools/diff_test_nf.py result/performance/cdm/fasttext.csv \
    -p fasttext -o result/
 
# All CDM results in a directory
python code/tools/diff_test_nf.py result/performance/cdm/ \
    -p all_cdm -o result/

Compute accuracy and sum difference for CDM:

python code/tools/compute_stat.py result/performance/cdm/fasttext.csv

Compute accuracy and sum difference for all baselines:

python code/tools/compute_stat_baseline.py \
    --bleu result/performance/baseline/bleu.csv \
    --bert result/performance/baseline/bert.csv \
    --sensim result/performance/baseline/sensim.csv \
    --dn result/performance/baseline/dn.csv

---

License

This project is released under the MIT License.

---