README
Bayesian-Style Document Distillation into a Frozen Language Model
This project explores whether a frozen language model can acquire current, document-grounded knowledge through small, incremental LoRA updates.
A document-conditioned teacher sees a source document and produces an informed next-token distribution. The student does not see the document; it represents the baseline, or prior, prediction. Training distills the difference between those predictions into a domain-specific LoRA adapter, allowing the student to answer later without receiving the source document in its prompt.
This is Bayesian-style posterior-predictive distillation, not exact Bayesian inference over model weights.
Experiment
The initial notebook pilot uses 500 CNN/DailyMail articles and follows this loop:
- Sample source documents from the dataset.
- Create grounded questions, canonical answers, and held-out paraphrases.
- Have the teacher answer with the relevant document in context.
- Record token-level teacher and student targets as an append-only update log.
- Train a LoRA adapter while keeping the student model frozen.
- Evaluate the adapter on paraphrased questions without the document.
The update log is intended to preserve document provenance, probe data, token-level targets, and evidence weights so updates can be replayed, reweighted, audited, and periodically compiled into an adapter.
Questions Being Tested
- Knowledge acquisition: Does the distilled LoRA answer questions about processed documents more accurately than the frozen student?
- Paraphrase transfer: Does that improvement hold for semantically equivalent questions that were not used to create training targets?
- Soft-target value: Do teacher probability distributions outperform ordinary supervised fine-tuning on hard answer tokens?
- Continual retention: Can sequential document updates be accumulated with limited forgetting using replay and preservation losses?
- Evidence sensitivity: Do repeated, contradictory, duplicate, and weighted sources produce predictable changes in the learned update?
- Efficiency: What accuracy, latency, and storage tradeoff does an adapter offer compared with passing full documents through retrieval at inference?
Pre-Registered Evaluation Figures
| Figure | Target Hypothesis | Comparison Conditions | Validation Outcome (Pass) | Falsification Outcome (Fail) |
|---|---|---|---|---|
| Figure 1: Acquisition & Transfer | H1 (Acquisition) & H2 (Transfer) | Frozen Base vs. trained LoRA vs. document-context reference on $Q_{train}$ and held-out wordings | LoRA gains $\ge +0.50$ F1 on $Q_{train}$ and retains $\ge 70%$ of its direct-question F1 on paraphrases without document context | Either threshold fails; inspect per-example results for memorization or prompt sensitivity |
| Figure 2: Soft vs. Hard Distillation | H3 (Soft Bayesian Targets) | Frozen Prior vs. Hard SFT vs. Soft Bayesian KL Distillation ($T \in {1, 2}$) | Soft targets yield higher paraphrase F1 and lower calibration error than Hard SFT | Hard SFT matches or outperforms Soft KL (teacher logit geometry adds no signal) |
| Figure 3: Continual Retention | H4 (Catastrophic Forgetting) | Sequential batches ($t=1\dots N$) with & without replay buffer | Old knowledge retention stays flat ($\ge 85%$) as new documents are ingested | Old accuracy collapses as new documents are learned |
| Figure 4: Evidence Sensitivity | H5 (Belief Revision) | Single source vs. 3 corroborating sources vs. 1 contradiction | Monotonic confidence increase with corroboration; weighted revision with contradiction | Adapter saturates immediately or fails to update on contradictory facts |
| Figure 5: Pareto Frontier | H6 (Efficiency vs. RAG) | Distilled LoRA vs. In-Context RAG (Accuracy vs. Latency / VRAM) | LoRA matches $\ge 85%$ of RAG accuracy with $5\times\text{--}10\times$ lower latency | RAG overwhelmingly dominates in both accuracy and practicality |
Project Layout
- teach-stu-bayes-update-news.py: Main standalone script with training loop, Oracle baseline, and Figure 1 & 2 generators.
- Teacher_Student_Bayesian_Token_Updates.ipynb: Interactive notebook pilot.
- pyproject.toml: Python environment and model dependencies.
Running the Experiment
The project targets Python 3.13 and uses uv for its environment:
uv sync
uv run python teach-stu-bayes-update-news.py
The five-document path is a pipeline smoke test, not a hypothesis-confirming
experiment. It currently trains a hard-answer SFT LoRA; soft-target/Bayesian
distillation requires a separate implementation and must not be inferred from
Figure 1. Each run writes auditable predictions, per-example scores, aggregate
scores, and the preregistered gate decision to
smoke_results/smoke_metrics.json.
Contributors
10 commits
Similar repos
Languages
Jupyter Notebook
61.4%
Python
38.6%
README
Bayesian-Style Document Distillation into a Frozen Language Model
This project explores whether a frozen language model can acquire current, document-grounded knowledge through small, incremental LoRA updates.
A document-conditioned teacher sees a source document and produces an informed next-token distribution. The student does not see the document; it represents the baseline, or prior, prediction. Training distills the difference between those predictions into a domain-specific LoRA adapter, allowing the student to answer later without receiving the source document in its prompt.
This is Bayesian-style posterior-predictive distillation, not exact Bayesian inference over model weights.
Experiment
The initial notebook pilot uses 500 CNN/DailyMail articles and follows this loop:
- Sample source documents from the dataset.
- Create grounded questions, canonical answers, and held-out paraphrases.
- Have the teacher answer with the relevant document in context.
- Record token-level teacher and student targets as an append-only update log.
- Train a LoRA adapter while keeping the student model frozen.
- Evaluate the adapter on paraphrased questions without the document.
The update log is intended to preserve document provenance, probe data, token-level targets, and evidence weights so updates can be replayed, reweighted, audited, and periodically compiled into an adapter.
Questions Being Tested
- Knowledge acquisition: Does the distilled LoRA answer questions about processed documents more accurately than the frozen student?
- Paraphrase transfer: Does that improvement hold for semantically equivalent questions that were not used to create training targets?
- Soft-target value: Do teacher probability distributions outperform ordinary supervised fine-tuning on hard answer tokens?
- Continual retention: Can sequential document updates be accumulated with limited forgetting using replay and preservation losses?
- Evidence sensitivity: Do repeated, contradictory, duplicate, and weighted sources produce predictable changes in the learned update?
- Efficiency: What accuracy, latency, and storage tradeoff does an adapter offer compared with passing full documents through retrieval at inference?
Pre-Registered Evaluation Figures
| Figure | Target Hypothesis | Comparison Conditions | Validation Outcome (Pass) | Falsification Outcome (Fail) |
|---|---|---|---|---|
| Figure 1: Acquisition & Transfer | H1 (Acquisition) & H2 (Transfer) | Frozen Base vs. trained LoRA vs. document-context reference on $Q_{train}$ and held-out wordings | LoRA gains $\ge +0.50$ F1 on $Q_{train}$ and retains $\ge 70%$ of its direct-question F1 on paraphrases without document context | Either threshold fails; inspect per-example results for memorization or prompt sensitivity |
| Figure 2: Soft vs. Hard Distillation | H3 (Soft Bayesian Targets) | Frozen Prior vs. Hard SFT vs. Soft Bayesian KL Distillation ($T \in {1, 2}$) | Soft targets yield higher paraphrase F1 and lower calibration error than Hard SFT | Hard SFT matches or outperforms Soft KL (teacher logit geometry adds no signal) |
| Figure 3: Continual Retention | H4 (Catastrophic Forgetting) | Sequential batches ($t=1\dots N$) with & without replay buffer | Old knowledge retention stays flat ($\ge 85%$) as new documents are ingested | Old accuracy collapses as new documents are learned |
| Figure 4: Evidence Sensitivity | H5 (Belief Revision) | Single source vs. 3 corroborating sources vs. 1 contradiction | Monotonic confidence increase with corroboration; weighted revision with contradiction | Adapter saturates immediately or fails to update on contradictory facts |
| Figure 5: Pareto Frontier | H6 (Efficiency vs. RAG) | Distilled LoRA vs. In-Context RAG (Accuracy vs. Latency / VRAM) | LoRA matches $\ge 85%$ of RAG accuracy with $5\times\text{--}10\times$ lower latency | RAG overwhelmingly dominates in both accuracy and practicality |
Project Layout
- teach-stu-bayes-update-news.py: Main standalone script with training loop, Oracle baseline, and Figure 1 & 2 generators.
- Teacher_Student_Bayesian_Token_Updates.ipynb: Interactive notebook pilot.
- pyproject.toml: Python environment and model dependencies.
Running the Experiment
The project targets Python 3.13 and uses uv for its environment:
uv sync
uv run python teach-stu-bayes-update-news.py
The five-document path is a pipeline smoke test, not a hypothesis-confirming
experiment. It currently trains a hard-answer SFT LoRA; soft-target/Bayesian
distillation requires a separate implementation and must not be inferred from
Figure 1. Each run writes auditable predictions, per-example scores, aggregate
scores, and the preregistered gate decision to
smoke_results/smoke_metrics.json.
Similar repos
Contributors
10 commits
Languages
Jupyter Notebook
61.4%
Python
38.6%