SDTD — Soft Contamination & Test Data

Code accompanying the paper Soft Contamination Means Benchmarks Test Shallow Generalization (anonymous, under review).

End-to-end research codebase for measuring soft contamination of LLM training data by semantic duplicates of benchmark test items, and for measuring how that contamination inflates downstream benchmark scores. Built around an OLMo3 case study.

Headline findings the code reproduces:

• Semantic duplicates of CodeForces items appear in OLMo3's training corpus for 78% of problems; exact duplicates of ZebraLogic for 50%.
• Fine-tuning OLMo3 on semantic duplicates of held-out benchmark items also improves performance on truly held-out items from the same benchmark — i.e. shallow, within-benchmark generalization rather than capability growth.
• An ecologically calibrated mix of semantic duplicates injected into Dolci-Instruct training data raises MuSR scores by ~5%.

---

Contents

• Quick start
• Project layout
• Reproducing the paper
• Data & secrets
• Building a code-only zip
• License & citation

---

Quick start

Requires Python ≥ 3.11 and uv.

# 1. Clone and enter
git clone <this-repo> SDTD && cd SDTD

# 2. Create the env and install
uv sync

# 3. Activate
source .venv/bin/activate              # macOS/Linux
.venv\Scripts\Activate.ps1             # Windows PowerShell

# 4. Set API keys you need (see "Data & secrets" below)
cp .env .env.local && $EDITOR .env.local   # then export from .env.local

Optional one-shot installers: setup.sh (macOS/Linux), setup.ps1 (Windows). Optional dev extras: uv pip install -e ".[dev]". Flash-attn (CUDA, Linux): uv pip install flash-attn --no-build-isolation.

Sub-projects with their own dependency specs (sdtd-llm-generation/pyproject.toml, finetuning/requirements.txt, pipeline/requirements.txt) are documented in their own READMEs.

---

Project layout

SDTD/
├── sdtd-llm-generation/   Generate semantic duplicates of benchmark items (LLM-driven)
├── pipeline/              6-stage embedding pipeline: scan OLMo3 corpora for high-cosine matches
├── comparison_analysis/   Sample · annotate (Gemini) · refine · classify · plot the matches
├── model_diffusion/       Tinker LoRA training of duplicate-detection classifiers
├── finetuning/            Controlled fine-tuning on exact / semantic / cosine-neighbor duplicates
├── ecology/               Ecologically valid contamination experiments + statistical tests
├── analysis_scripts/      Supporting tools: Anthropic-API analysis, S3 match resolution, dedup
├── utils/                 Shared helpers: embedding mean-pool, n-gram similarity
├── pyproject.toml · uv.lock · setup.{sh,ps1}
├── make_submission_zip.sh
└── LICENSE · README.md

Every directory has a README.md. The two largest (comparison_analysis/, ecology/) are split into per-stage subdirs (e.g. annotation/, classifier/, plots/, data_prep/, training/, evaluation/), each with its own README.

---

Reproducing the paper

Each block is the entry point for one experiment; subdirectory READMEs document configuration, hardware requirements, and outputs in detail.

1 · Generate semantic duplicates · §3.1

cd sdtd-llm-generation && uv run python -m sdtd generate --dataset mbpp --level 1

2 · Scan training corpora · §3.2

bash pipeline/run_all.sh

3 · Annotate + classify candidate matches · §3.2, §4.2

python comparison_analysis/annotation/sample_for_annotation.py     --benchmark mbpp
python comparison_analysis/annotation/annotate_semantic_duplicates.py --benchmark mbpp
python model_diffusion/train_mbpp_classifier.py
python comparison_analysis/plots/generate_all_plots.py

4 · Controlled fine-tuning · §4.3

accelerate launch --num_processes 8 finetuning/mbpp/train_and_eval.py
python finetuning/musr/finetune_musr_mixed.py

5 · Ecologically valid contamination · §4.4

python ecology/data_prep/sample_dolci.py
python ecology/data_prep/create_contaminated_dataset.py
python ecology/training/run_experiment_tinker.py --model meta-llama/Llama-3.1-8B
python ecology/evaluation/evaluate_contamination.py

---

Data & secrets

No data ships with this repo. Pre-training and benchmark datasets must be downloaded from their original sources:

API keys are read from environment variables. .env is gitignored — copy the placeholder file and fill in only the keys you need:

---

Models & libraries

• Embedding model for §3.2 corpus scan: nvidia/llama-embed-nemotron-8b
• Models evaluated: OLMo-3-1025-7B, Qwen3-8B-Base, Qwen3.5-9B-Base, Llama-3.1-8B (Tinker path supports any HF base via --model)
• Adaptation: QLoRA via PEFT/TRL; remote LoRA training via Tinker API
• SD generation backends: Claude, GPT, Qwen via OpenRouter
• Benchmarks: MBPP, HumanEval, CodeForces, MuSR (Murder Mysteries, Team Allocation), ZebraLogic

---

Building a code-only zip

bash make_submission_zip.sh

Produces ../SDTD_submission.zip. The script:

• includes only source code, configs, and READMEs
• excludes .env, virtualenvs, datasets, embeddings, model checkpoints, generated figures, logs, and caches
• refuses to run if .env still holds a non-placeholder API key

---

License & citation

MIT License.

@inproceedings{anonymous2026softcontamination,
  title  = {Soft Contamination Means Benchmarks Test Shallow Generalization},
  author = {Anonymous},
  booktitle = {International Conference on Machine Learning},
  year   = {2026}
}