Quality-Gated Distillation — experiment and analysis code

Code, configuration files, and the evidence ledger behind:

Quality-Gated Distillation for Efficient Spatio-Temporal Forecasting: When (and When Not) to Trust a Foundation Teacher Ardi Yusri Hilmi, Edi Winarko, Moh Edi Wibowo, Andi Dharmawan Department of Computer Science and Electronics, Universitas Gadjah Mada

Regenerating the analysis artifacts reproduces the paper's gate table on all nine datasets and the headline correlation at r = 0.963 (Pearson, validation split, n = 9). Read the split label before comparing that number against anything: the same correlation on the test split is r = 0.991, and earlier drafts of this file quoted the test figure while the paper quoted the validation one.

---

What this repository is for

The paper argues that a foundation teacher should supervise a compact student only after passing an empirical reliability gate, and that silent cache and configuration errors are a first-order source of misleading distillation results. Three of the paper's own numbers had to be withdrawn and recomputed because of exactly such errors.

That argument only holds up if the numbers can be traced. So this repository is organised around one question: for any table or figure in the paper, which script produced it and from what input? The map below answers that for every artifact.

---

From paper artifact to script

Table numbers follow the MethodsX submission. One command rebuilds every CSV below from the vendored artifacts:

python3 analysis/scripts/rebuild_ledger.py

Table 1 is a description of the datasets and Table 8 is a narrative account of the three provenance faults; neither is computed. Figures 1 and 2 are schematics drawn in the manuscript source. Figure 3 is plotted from the same gate evidence as Table 2 and Figure 4 from the same grid as Table 6.

The zero-shot candidate evaluations that feed Table 7 are run by evaluate_moirai.py and evaluate_timesfm.py, which write results/{moirai,timesfm}_<dataset>.json; rebuild_ledger.py reads those and the vendored traffic screening into one table.

One column this repository cannot reproduce

The Chronos-2 entries for the five energy datasets in Table 7 come from a context-12 run whose per-dataset outputs were never exported from the training host. What is archived here under analysis/results/upstream/tsfm/ is the context-168 benchmark, which is a different configuration and gives different numbers. cross_family.csv marks those five cells NOT VENDORED rather than quietly substituting the context-168 values, and the paper's data availability statement says the same. Everything else in the table traces to a file here.

---

Two things a reader should know before trusting an output

1. The raw experimental results are inherited, not computed here. The analysis layer derives the paper's tables from the run artifacts under analysis/results/upstream/, chiefly the seven rev1_batch*.json files that record the revision reruns: the matched-config five-seed KD comparison (rev1_batchG), the post-adjacency-fix teacher retrain for ACN-EV and OPSD-Wind (rev1_batchA_graphfix), the gate statistics on both splits (rev1_batchA_corrected), the α/β grid and traffic screening (rev1_batchC), and the task-trained baselines (rev1_batchB). Those come in turn from the training code in src/. The ST-Chronos tree they were taken from is tagged paper3-qgd-source (commit 2302e8b), so the upstream revision is named rather than implied.

2. A shipped artifact can be stale, and one was. The first version of this repository built its gate evidence with build_mlj_artifacts.py, which read teacher nMAE from a hardcoded table. The ACN-EV entry in that table sat one correction behind the paper: it held the value from a teacher trained on a pre-fix adjacency graph, and eight of nine rows of the shipped statistical_tests.csv disagreed with the published effect table, two of them in sign. That is the failure the paper itself is about, reproduced in its own supplementary code.

The fix was to stop deriving the ledger from a hardcoded snapshot. rebuild_ledger.py now assembles every CSV from the revision artifacts vendored under analysis/results/upstream/revision_artifacts/, each row carrying the artifact it came from in a source column, and the superseded July generation is kept under analysis/results/superseded_2026-07/ rather than deleted so the two can be compared. docs/RELEASE_BLOCKERS.md has the full account.

---

Layout

configs/          experiment configuration
src/models/       teacher, students, and task-trained baselines
src/utils/        dataset loading, metrics, checkpoint bookkeeping
src/scripts/      dataset download, training, evaluation
analysis/scripts/ gate, statistics, and figure generation
analysis/results/ the small CSV/JSON artifacts each number traces to
tests/            unit tests
docs/             copy manifest and provenance notes

Not included

Raw datasets (~2.3 GB) and trained checkpoints (~539 MB) are not distributed here. src/scripts/download_*.py fetches the datasets; training scripts regenerate the checkpoints. Each dataset's source is cited in the paper's Data Availability Statement.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Then rebuild the ledger and run the tests:

python3 analysis/scripts/rebuild_ledger.py
python3 -m pytest tests/ -q

rebuild_ledger.py is deterministic: on a clean checkout it rewrites the CSVs byte-identically, so git status stays empty. One test skips because it reads a training artifact that is not distributed here.

Citation

See CITATION.cff. The DOI is added once the method article is published; until then cite the preprint metadata in that file.

License

MIT, see LICENSE. The manuscript itself is published open access under CC BY.

Contact

Ardi Yusri Hilmi — ardi.y.h@mail.ugm.ac.id Edi Winarko (corresponding author) — ewinarko@ugm.ac.id