Sfumato — source repo for the E2/E4 hybrid AR / DDLM experiments

This repo holds the experiment harness, training scripts, datasets, adapters, and results for the paper

Three Orthogonal Failure Axes in Hybrid AR / Diffusion Reasoning, with Trainable Fixes for Two of Them

The LaTeX source of the paper, the rendered PDF, and the Manim video live in a separate sibling repo at https://github.com/eren23/sfumato_paper.

What's here

e4/                AR vs diffusion vs hybrid CoT runner (the actual eval harness)
  runner.py        env-var-driven entry point; CONDITION = c1 | c2 | c2c | c3 | c3p | c4 | cmaj | cmajc | cmerge
  diff_llada.py    LLaDA-8B-Instruct loader + semi-AR denoise loop with optional commit-LoRA
  ar_qwen.py       Qwen2.5-{0.5B,1.5B}-Instruct planner / finalizer
  grade.py         numeric-answer regex extractor + scoring
  flops.py         FLOPs accounting per condition
  data/            frozen GSM8K-test problem indices (gsm8k_dev_200.json)
  results/         per-run JSONL outputs

e2/                Paper 1 work record
  PROTOCOL.md      pre-registered hypotheses, success/kill criteria
  RESULTS_TRACK1.md  Track 1 (prefix-robust LoRA) hyperparameter ablations + eval
  RESULTS_TRACK2.md  Track 2 (commit-LoRA) design iteration v1->v2->v3 + ablations
  PAPER_DRAFT.md   working markdown draft (LaTeX is canonical; lives in sfumato_paper)
  figs/            paper figures (regenerated by scripts/make_paper_figures.py)
  data/            local cached datasets (gitignored; source of truth on HF Hub)
  results/         per-condition raw eval JSONLs

scripts/
  train_track1_lora.py        Track 1 prefix-robust LoRA trainer
  train_track2_commit.py      Track 2 commit-LoRA trainer
  build_prefix_robust_dataset.py  builds eren23/sfumato-prefix-robust-gsm8k (8 prefix tiers x 7,473 problems)
  build_consensus_dataset.py     builds eren23/sfumato-consensus-gsm8k (500 cmaj b=5 outputs)
  build_commit_mixture.py        builds eren23/sfumato-commit-mixture-gsm8k (109-row training mixture)
  branch_agreement.py         analyses branch-vote diversity (Figure 2 numbers)
  commit_effect_diagnostic.py greedy text-shift / answer-flip comparison at temp=0
  eval_planner_quality.py     planner ablation across {Q-0.5B, Q-1.5B, Q-7B, oracle gold}
  make_paper_figures.py       regenerates Figures 1-4 from hard-coded numbers
  binom_ci.py                 Clopper-Pearson 95% CI helper for paper tables
  abl_b_sanity.py             5-problem text-diff probe verifying ABL_B adapter loads (Phase C)
  PHASE_C_RUN.md              cmajc-v3 reproduction guide (env-vars + decision rules)
  analyze_e4.py, freeze_gsm8k.py, inspect_run.py, ...   misc helpers

crucible.yaml      Crucible local-run config for E4
.gitignore         skips wandb/, .venv/, e2/data/, .claude/, .attocode/

Running experiments

A. Fleet (Crucible + RunPod, the way the paper was produced)

The paper's experiments dispatch through Crucible Fleet onto a single RTX 4090 spot pod. The project spec lives at <parameter-golf_dev>/.crucible/projects/sfumato_e4.yaml; this repo just exposes the runner and crucible.yaml.

# Crucible MCP call (or analogous CLI):
run_project(
    project_name="sfumato_e4",
    overrides={
        "CONDITION": "cmajc",           # c1|c2|c2c|c3|c3p|cmaj|cmajc|...
        "K_STEPS": "64",                # diffusion denoising steps
        "N_PROBLEMS": "200",
        "SEED": "0",
        "LORA_PATH": "eren23/sfumato-llada-prefix-robust-v3",
        "COMMIT_LORA_PATH": "eren23/sfumato-llada-commit-v3",
        "COMMIT_N_BLOCKS": "3",         # commit on sub-blocks 2-4 of 4
        "BRANCHES": "5",                # cmaj/cmajc branches
        "TEMP": "0.7",
        "WANDB_RUN_NAME": "cmajc-v3-N200",
    },
)

Reproducing the paper's headline cmajc-v3 = 82.5% number end-to-end: see scripts/PHASE_C_RUN.md.

B. Local (no fleet)

git clone https://github.com/eren23/sfumato.git
cd sfumato
python -m venv .venv && source .venv/bin/activate
pip install transformers==4.46.3 peft accelerate datasets huggingface_hub wandb numpy scipy

# Smoke test (no GPU, mock weights):
.venv/bin/crucible run experiment --preset smoke

# Real eval (needs RTX 4090 / similar):
CONDITION=cmajc K_STEPS=64 N_PROBLEMS=200 SEED=0 \
  LORA_PATH=eren23/sfumato-llada-prefix-robust-v3 \
  COMMIT_LORA_PATH=eren23/sfumato-llada-commit-v3 \
  COMMIT_N_BLOCKS=3 BRANCHES=5 TEMP=0.7 \
  python e4/runner.py

The runner emits LM-training-contract lines that Crucible parses: step:I/N train_loss:<1-acc> val_loss:<1-acc> val_bpb:<flops>. The final accuracy lands in W&B and in e4/results/.

MOCK_MODELS=1 runs the harness without loading real weights — useful for verifying the Crucible parser path before paying for a GPU.

C. Local inference with the trained hybrid (cmajc-v3 / c2c / c3)

"Hybrid training" = the two LoRAs we trained on top of LLaDA-8B-Instruct (Track 1: prefix-robust-v3, Track 2: commit-v3) composed with Qwen2.5 into the cmajc / c2c / c3 conditions. Same e4/runner.py entry point — the env vars decide whether the trained adapters are loaded.

Hardware constraints. Real-mode LLaDA-8B requires CUDA (no MPS path). On a Mac/CPU the .to("cuda") falls back to fp32 CPU and will OOM at ~32 GB resident or run unusably slow. Use MOCK_MODELS=1 on the laptop; provision a 4090 (Crucible/RunPod, $0.20–0.34/h) for real generation.

Setup once:

python3 -m venv .venv && source .venv/bin/activate
pip install transformers==4.46.3 peft accelerate datasets huggingface_hub wandb numpy scipy
export HF_TOKEN=hf_...   # only needed if any LoRA repo is private

Mock smoke (laptop, ~5s, validates the whole hybrid plumbing without loading weights — adapters are name-checked but not downloaded):

MOCK_MODELS=1 CONDITION=cmajc K_STEPS=64 N_PROBLEMS=5 SEED=0 \
  LORA_PATH=eren23/sfumato-llada-prefix-robust-v3 \
  COMMIT_LORA_PATH=eren23/sfumato-llada-commit-v3 \
  COMMIT_N_BLOCKS=3 BRANCHES=5 TEMP=0.7 \
  python e4/runner.py

Real cmajc-v3 (the headline 82.5% recipe; CUDA ≥24 GB):

WANDB_DISABLED=1 CONDITION=cmajc K_STEPS=64 N_PROBLEMS=5 SEED=0 \
  LORA_PATH=eren23/sfumato-llada-prefix-robust-v3 \
  COMMIT_LORA_PATH=eren23/sfumato-llada-commit-v3 \
  COMMIT_N_BLOCKS=3 BRANCHES=5 TEMP=0.7 \
  python e4/runner.py
# bump N_PROBLEMS=200 for the full dev split

Single-shot variants (cheaper, same adapters):

# c2c — diffusion + commit-LoRA only (no branching, no AR):
CONDITION=c2c K_STEPS=64 N_PROBLEMS=5 SEED=0 \
  LORA_PATH=eren23/sfumato-llada-prefix-robust-v3 \
  COMMIT_LORA_PATH=eren23/sfumato-llada-commit-v3 \
  COMMIT_N_BLOCKS=3 python e4/runner.py

# c3 — AR plan -> diffusion CoT -> AR finalize (no commit):
CONDITION=c3 K_STEPS=64 N_PROBLEMS=5 SEED=0 \
  LORA_PATH=eren23/sfumato-llada-prefix-robust-v3 \
  python e4/runner.py

Token-by-token visualizer (live grid, manual-mode buttons at every sub-block boundary): see phase2/inference_viz/LOCAL_QUICKSTART.md. Mock mode runs locally, real mode tunnels to a pod.

Key conditions

• c1 — pure AR (Qwen alone)
• c2 — pure diffusion (LLaDA alone, no prefix)
• c2c — c2 with commit-LoRA enabled (last sub-block for v2; sub-blocks 2-4 for v3)
• c3 / c3p — AR plan -> LLaDA-conditioned-on-plan -> answer extraction
• c4 — c3 + extra AR-extend / diffuse-again round
• cmaj (b=5, t=0.7) — 5 stochastic LLaDA branches, majority vote on extracted answers
• cmajc — cmaj with commit-LoRA enabled per branch, then majority vote
• cmerge — diffusion-germinate-multiple -> AR-converge-into-one

Pre-registered predictions

See e2/PROTOCOL.md. The four predictions and their final state:

Artifacts

Public on the Hugging Face Hub:

• eren23/sfumato-llada-prefix-robust-v2 — Track 1, 4/7-module LoRA (10M params)
• eren23/sfumato-llada-prefix-robust-v3 — Track 1, 7/7-module LoRA (22M params, headline)
• eren23/sfumato-llada-commit-v2 — Track 2, last-block answer-span LoRA (13.6M params)
• eren23/sfumato-llada-commit-v3 — Track 2, multi-block full-response LoRA (13.6M params, headline)
• eren23/sfumato-prefix-robust-gsm8k — 7,473 GSM8K-train problems x 8 prefix tiers (59,784 rows)
• eren23/sfumato-consensus-gsm8k — 500 GSM8K-train cmaj b=5 outputs
• eren23/sfumato-commit-mixture-gsm8k — 109-row training mixture (rescue / preserve-disagreement / pure-agreement)

W&B: https://wandb.ai/eren23/sfumato-e2, https://wandb.ai/eren23/sfumato-e4.

Pinned dependencies

Pin transformers==4.46.3. Newer transformers releases break LLaDA's LLaDALlamaBlock loader. See e4/diff_llada.py:120.

Total compute spend

~$3.50 across the entire paper, single RTX 4090 spot pod at $0.20/hr on RunPod. Phase C (cmajc-v3 + ABL_B sanity) added ~$1.00 of revision compute. Cost breakdown in paper/appendix/compute_spend.tex (sfumato_paper repo).

Paper

The paper, video, and built PDF live at https://github.com/eren23/sfumato_paper. This repo is the source for the experiments referenced there.

Prior work

• First CodeWM preprint: https://eren23.github.io/codewm-paper-public/
• Second CodeWM preprint: https://eren23.github.io/codewm2-paper-public/
• Third CodeWM preprint (CodeDeltaTok): https://github.com/eren23/codewm3-paper

License

MIT.