The bias is in the generator

Code for the paper "The bias is in the generator: locating medical chain-of-thought subgroup gaps upstream of the verifier" (AACL-IJCNLP 2026, Main Conference).

The paper asks where a subgroup accuracy gap comes from in verifier-guided best-of-N decoding: from the generator (correct answers are sampled less often for one subgroup) or from the verifier (it selects differently between subgroups). This repository contains everything needed to rerun the experiments and to recompute every number, table and figure in the paper.

Contents

1. Main findings
2. What is in this repository
3. Setup
4. Recompute the paper's numbers from run outputs (CPU)
5. Regenerate the runs (GPU)
6. Where each table and figure comes from
7. Conventions
8. What to expect when you rerun
9. Repository layout · Citation · License

Main findings

All results use AMQA, a benchmark of adversarial demographic counterfactuals of USMLE-style questions (white vs. black, male vs. female, high vs. low income).

• The gap is already in the generator. On HuatuoGPT-o1-8B, the race gap (white minus black accuracy) in a single sampled answer (N=1) is 21.0 pp in the primary run and 16.6 pp on average over five seeds, before any verifier is applied.
• Verifiers rarely amplify it. No verifier significantly increases the gap on HuatuoGPT-o1-8B after Holm correction; across seven generators, only Gemma-3-27B-it under Skywork-RM does. Even an oracle that always picks a correct candidate when one exists leaves an 11.0 pp gap, so correct answers are less often available for one subgroup.
• Fixing the generator helps more than fixing the selector. A LoRA fine-tune on 401 counterfactually paired examples closes 46.4% of the N=1 gap on HuatuoGPT-o1-8B and 58.1% on Med42-8B (on HuatuoGPT-o1-8B the reduction does not persist after self-judge selection at N=16). Step-matched single-variant controls move the gap in opposite directions. A DPO variant also narrows the gap while keeping the model's chain of thought.

What is in this repository

No real patient data is used: every dataset is made of examination vignettes.

Setup

1. Python environment

conda create -n vpb python=3.12
conda activate vpb
pip install -r public/requirements.txt

public/requirements.txt lists minimum versions. The paper's runs used NVIDIA's PyTorch 25.10 container (torch 2.9.0a0) with transformers 5.5.0 and peft 0.13.2; install these exact versions (pip install transformers==5.5.0 peft==0.13.2). Newer transformers releases remove the warmup_ratio training argument that public/training/lora_train.py uses, and can change chat templates and tokenization (see What to expect when you rerun).

2. Datasets

Important: set AMQA_FILE. The code stops if AMQA_FILE is not set. Use the human-reviewed file above, not the copy on Hugging Face (Showing-KCL/AMQA), which is the unrevised release and gives different numbers.

3. Models

Some models are gated on Hugging Face: accept their terms before downloading (meta-llama/Llama-3.1-8B-Instruct, google/gemma-3-27b-it, google/medgemma-27b-it, epfl-llm/meditron-7b).

Some job templates load local copies through MODEL_DIR (for example $MODEL_DIR/Llama3-Med42-70B).

4. Environment variables

export AMQA_FILE=/path/to/AMQA_Dataset.jsonl   # required (see above)
export OUTPUTS_DIR=/path/to/outputs            # where runs are written and read; default: <repo>/outputs
export HF_HOME=/path/to/hf_cache               # Hugging Face cache
export MODEL_DIR=/path/to/local/models         # only for templates that load local model copies

Recompute the paper's numbers from run outputs (CPU)

Once the runs exist under OUTPUTS_DIR, one command recomputes every number in the paper and prints it next to the printed value with MATCH or MISMATCH. Topics whose inputs are missing are reported as SKIP; the command exits with an error if anything mismatches or is skipped.

bash public/reproduce.sh                         # all topics (about ten minutes on CPU)
bash public/reproduce.sh --quick                 # skip the slow bootstraps
bash public/reproduce.sh --list                  # list the topics
bash public/reproduce.sh --only locus,recipes    # run selected topics

Each topic is a module in public/paper/<topic>/ and can also be run alone, for example PYTHONPATH=. python -m public.paper.locus.locus.

Redraw all figures with:

bash public/paper/figures/make_figures.sh

Where outputs go. Run directories sit directly under OUTPUTS_DIR (for example outputs/p4_amqa/, outputs/p4_amqa_seed1/). Outputs of the scripts in public/paper/<topic>/ go to OUTPUTS_DIR/paper/<topic>/; the five-seed fairness, Med42-8B and transfer runs go to OUTPUTS_DIR/paper/five_seed/. Each topic directory can be redirected with PAPER_<TOPIC>_DIR (see public/paper/common.py). Three topics write derived files (holm, tagger_and_strata, medmcqa, into PAPER_HOLM_DIR, PAPER_TAGGER_DIR and PAPER_MEDMCQA_WORK_DIR); point these at a scratch directory to keep the run outputs untouched. Runs kept elsewhere can be found through OUTPUTS_FALLBACK_DIRS (colon-separated directories searched after OUTPUTS_DIR).

Regenerate the runs (GPU)

All runs used H100 80 GB GPUs: one GPU per job, two for the 70B generators. LoRA training needs a full 80 GB GPU because the base model is loaded in fp32; one training run takes four to eight minutes. A generation run (400 pairs × 16 samples, then scoring) fits within the 6 to 8 hours the templates request.

Submit jobs from the repository root. Scheduler lines (account, partition, QOS) are site-specific; each template marks where to put yours. Every template sources public/slurm/env.sh, which activates your environment (set ENV_SETUP, for example ENV_SETUP="conda activate vpb") and puts the repository on PYTHONPATH.

See public/slurm/README.md for the full list of variables the templates read.

Where each table and figure comes from

Run from the repository root with PYTHONPATH=.. Every python -m public.paper... command prints the recomputed values with MATCH / MISMATCH. Numbering follows the final paper: Tables 1–7 and Figures 1–4 in the main text, Tables A1–A18 and Figures A1–A4 in the appendix.

Main text

Appendix

Conventions

• Seeds. The run with sampling seed 0 is the primary run of each generator, dataset and axis; the verifier analyses (Sections 4.1–4.2), the multi-generator comparison and the dataset probes use it. Intervention results are means over five runs with seeds 1 to 5 (training and sampling seed), compared with the mean of five baseline runs with the same seeds. Standard deviations are sample s.d. over seeds.
• N=1 is the first of the 16 sampled candidates. Best-of-N picks the highest-scoring of the first N candidates; ties go to the earlier sample.
• Answer parser. public/data/load_eval.parse_answer is used everywhere: a leading answer letter, then "answer is X" and similar forms, then the last standalone A/B/C/D in the final 300 characters. Outputs without an answer count as incorrect.
• Flip rate is computed over pairs in which both variants give a parsed answer.
• Closure is 1 − gap(intervention) / gap(baseline), computed from unrounded values; the baseline is the same generator and axis (the primary run for single-run interventions). Values are rounded half up.
• Self-judge for LoRA models is the untuned base model: saved LoRA candidates are rescored with public/paper/base_judge/rescore_basejudge.sbatch.
• Verifier inputs contain the question stem and the candidate, not the answer options (run_p4.py, inference/score_with_rm.py). The free-text judges (Appendix J) do see the options.
• Sampling. Generation sets temperature 0.7, top-p 0.9 and 512 new tokens; all other sampling settings come from each model's generation_config.json and the transformers defaults.
• Duplicated BOS token. The self-judge, Skywork-RM and FRM inputs are built by applying the chat template and then tokenizing, which adds a second BOS token. All reported results use this input, except Appendix C's single-BOS rows (public/paper/bos_sensitivity/).

What to expect when you rerun

• Exact numbers will differ; conclusions should not. Generation samples at temperature 0.7 on GPU, which is not bit-for-bit reproducible across hardware and library versions. Individual runs vary noticeably: the baseline N=1 gap ranges from +0.128 to +0.228 across five seeds. Five-seed means and the paper's main effects are much more stable than single runs.
• Pin your versions. Model and dataset revisions on Hugging Face are not pinned in the code, and newer library versions can change chat templates and tokenization (the self-judge is sensitive to this; see Appendix C). Record the versions and model revisions you use.
• LLM-edited probes are regenerated, not replayed. The MedMCQA and LLM-edited race × sex counterfactuals are written by Gemma-3-27B-it at run time, so a rerun produces new edits. The MedMCQA faithfulness filter (public/paper/medmcqa/audit_labels.py) records the labels of the paper's own audit and applies only to the paper's edits.
• Re-submitted jobs. For R3 on HuatuoGPT-o1-8B seed 2, the first evaluation job wrote no output; the paper uses the re-submitted job (directory suffix _backup). For R1 seed 3, the first job wrote all 12,800 generations but stopped before scoring; the paper uses the first job's generations (both give the same N=1 gap; race mention is 35.4% vs. 35.6%). public/paper/common.resolve_candidates encodes this rule. The Med42-8B baseline of seed 3 stopped after self-judge scoring; its remaining steps were run on the saved candidates (public/paper/five_seed/resume_score.sbatch).
• Race × sex qids. The sex-word loader numbers kept pairs consecutively. The paper's women run was generated before one record whose options contradict the inserted sex ("gynecomastia") was excluded, so its qids skip one number; the analysis only compares qids within a run.
• MedQA-USMLE flip rates. The metrics behind Figure A1 were written before the flip rate was restricted to pairs where both answers parse; public/paper/flip_rates/patch_flip_metrics.py rewrites them.

Repository layout

public/
  reproduce.sh, reproduce.py   recompute and check every paper number (no GPU)
  run_p4.py                    best-of-N pipeline: generate, score, metrics
  paths.py                     OUTPUTS_DIR handling
  data/                        dataset loaders, counterfactual-pair construction, answer parser
  inference/                   verifier scoring, perturbation diagnostic, free-text generation and judging
  training/                    paired LoRA (SFT), R1 targets, R2 (KL), R3 (DPO)
  analysis/                    amplification tests, oracle / self-consistency, tagger, race mention, AUROC, figures
  paper/                       one folder per paper topic: collectors that print the paper's numbers,
                               and the job templates for that topic
  slurm/                       job templates for the pipeline runs (see slurm/README.md)

Citation

@inproceedings{nnamdi2026biasingenerator,
  title     = {The bias is in the generator: locating medical chain-of-thought subgroup gaps upstream of the verifier},
  author    = {Nnamdi, Micky C. and Marteau, Benoit Louis and Tan, Shaun Qien Yeau and Tamo, J. Ben and Yuan, Yining and Wang, May Dongmei},
  booktitle = {Proceedings of AACL-IJCNLP 2026},
  year      = {2026}
}

License

Code: MIT (see LICENSE). Datasets and models keep their own licenses.