MUDDLE

Data-construction and evaluation code for MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects, accepted (poster) at the COLM 2026 workshop on Context Beyond the Window.

Paper: https://openreview.net/forum?id=9YS7tSPM9c

What the benchmark does

MUDDLE separates two things that long-context and retrieval-noise studies tend to conflate: how far a distractor sits from the source in topic, and how much length it adds.

270 human-annotated questions, each tied to a single source document, are instantiated in five context conditions:

Random distractors are drawn from other questions' hard-negative pools, so the two arms match on provenance and length and differ only in topical relevance. Distractor selection is deterministic (seed 20260523), so every model consumes byte-identical cells.

Data

The documents themselves are on the Hugging Face Hub, not in this repo.

All four are CC BY 4.0. Every cell folder is self-contained: the documents in canonical position order (source first) plus a question.json with the cell metadata.

Layout

pipeline/   numbered construction scripts (1..16) plus the eval + judging code
reports/    curation, filtering, packing, and token-audit reports
results/    raw per-cell model outputs, one JSON object per line

Pipeline

The numbered scripts run in order and each writes an artifact the next one reads:

• 1..3 — pull MMLongBench-Doc, filter to usable questions, extract text
• 4..5 — token counting and the length audit that the length-matching depends on
• 6..7 — build the random-distractor and hard-negative pools
• 8 — pack every (question, condition) pair into cells
• 9 — run the evaluation (run.py, run_md.py, run_img.py per modality)
• 10..16 — analysis, hard-negative validation, and the retrieval/top-up passes used to fill gaps in the HN pool

Shared modules: pack.py, run.py, adapt.py (translate the PDF plan into markdown or image plans), judge.py / llm_judge.py (LLM-as-judge scoring), context_utils.py.

Results

results/*.jsonl hold one record per evaluated cell:

{
  "cell_id": "q0_random_k4",
  "model": "gpt-5-mini-low",
  "condition": "random",
  "k": 4,
  "question": "...",
  "answer": "...",
  "response": "...",
  "score_f1": 0.42,
  "score_em": 0.0,
  "judge_correct": true,
  "usage": {"in": 12345, "out": 42},
  "n_pages": 80,
  "est_tokens": 32000
}

The raw response is always kept, so the judge can be swapped without re-calling any model. These files are also what a per-item mixed-effects re-analysis with token count as a covariate would run against — the check the paper's Limitations section names as the one that would settle the residual length gap between the arms.

Reported scope

All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported in the paper is run in markdown: a source document plus its distractors exceeds current image and PDF input limits. The image and PDF renderings support the source-only comparison.

Configuration

The scripts read credentials from the environment and never hard-code them:

AZURE_OPENAI_API_KEY=
AZURE_OPENAI_ENDPOINT=
AZURE_OPENAI_DEPLOYMENT=
AZURE_OPENAI_API_VERSION=
OPENROUTER_API_KEY=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION_NAME=

Citation

@inproceedings{luo2026muddle,
  title     = {{MUDDLE}: Measuring Understanding of Documents under Distractor and Length Effects},
  author    = {Luo, Jason and Abudukelimu, Saibilila and Song, Judy and Feng, Andrew
               and Garg, Shivank and Sharma, Vasu and Zhu, Kevin},
  booktitle = {COLM 2026 Workshop on Context Beyond the Window},
  year      = {2026},
  url       = {https://openreview.net/forum?id=9YS7tSPM9c}
}

License

Code is MIT (see LICENSE). The released datasets are CC BY 4.0. Source documents come from MMLongBench-Doc and from public web retrieval, and remain under their original terms.