GLU — Global-Local Uncertainty for LLMs

Code for the paper "Integrating Local and Global Entropy for Uncertainty Quantification in LLMs".

GLU is a training-free uncertainty score that combines two complementary signals extracted from a single greedy generation:

• Local uncertainty — per-token Shannon-weighted epistemic uncertainty derived from Evidence Deep Learning (EDL), captured by she_R_mean (mean over the k worst-scoring tokens).
• Global geometric uncertainty — normalized Matrix Rényi entropy S̃ of the hidden-state trajectory across generated tokens, measuring how much the model's internal representation "spreads out" over the sequence.

The final score is:

GLU = (1 + S̃) * she_R_mean

she_R_mean is negative and (1 + S̃) ≥ 1, so GLU is always ≤ 0. A more negative GLU indicates higher uncertainty. AUROC is computed with GLU as the score against correctness labels (label = 1 means correct).

---

Repository structure

GLU/
├── generate_response.py   # Step 1 — run models, collect UQ features, judge correctness
├── GLU.py                 # Step 2 — AUROC of the baseline GLU score per model–dataset
├── GLU_ablations.py       # Step 2 — full ablation table across all GLU variants
├── .env.example           # Azure OpenAI credentials template (for the LLM judge)
├── small_datasets/
│   ├── triviaqa_small.csv
│   ├── truthfulqa.csv
│   ├── math_small.csv
│   ├── arabica_qa.csv
│   ├── longform_subset.csv
│   └── multiturn_math.csv       # not yet wired into generate_response.py
└── output/
    └── full/              # created automatically; all Step 1 outputs land here

---

Step 1 — Generate responses and compute UQ features

generate_response.py loads a model once, runs greedy generation on every dataset sample, and writes a JSONL file with all UQ features plus an LLM-judge correctness label.

Prerequisites

pip install torch transformers datasets openai scikit-learn scipy pandas tqdm python-dotenv numpy

Copy .env.example to .env and fill in your Azure OpenAI credentials:

cp .env.example .env

Usage

python generate_response.py \
    --model  <fanar|gemma|qwen> \
    --datasets <triviaqa|triviaqa_full|truthfulqa|math|arabica|longform> [<dataset2> ...] \
    [--max-new-tokens 256] \
    [--k 10] \
    [--alpha 2.0] \
    [--layer-idx -1] \
    [--seed 0] \
    [--rauq-alpha 0.2] \
    [--rauq-use-hooks] \
    [--max-samples N] \
    [--output-suffix "-v2"] \
    [--test]

Quick smoke test

python generate_response.py --model qwen --datasets triviaqa --test

Outputs

Each run writes two files to ./output/full/:

Rows are flushed as they are generated, so a partially finished run still leaves a valid JSONL file. Samples that fail generation are skipped and not written. label is null when the judge returns something other than 0/1.

The script also prints live AUROC figures at the end of each dataset.

Fields in each JSONL row

List and dict fields are stored as native JSON, so no extra decoding is needed. To load a file:

import pandas as pd
df = pd.read_json("output/full/qwen-triviaqa-unified-uq-full.jsonl", lines=True)

---

Step 2 — Compute GLU and ablations

Both scripts read every output/full/*-unified-uq-full.jsonl file produced by Step 1. Run them from the repo root. Rows with a null label are dropped before computing AUROC.

Files written with a non-empty --output-suffix (e.g. ...-unified-uq-full-v2.jsonl) don't match this pattern and are ignored.

GLU.py — single GLU score per file

Prints AUROC(GLU) per model–dataset pair and the number of dropped rows.

python GLU.py

GLU_ablations.py — full ablation table

Groups files by dataset and skips incomplete runs (files with fewer rows than the most complete run for that dataset). It then prints a table of AUROC scores for all GLU variants side by side.

python GLU_ablations.py

Ablation variants

---

Supported models and datasets