DIMER pipeline for Aurora 0.25° small: audited pickle conversion to safetensors, 6-hourly global forecasting against persistence, and bounded LoRA fine-tuning on pinned ERA5 windows
Jupyter Notebook
0
50 commits
updated Oct 14, 2026
DIMER-oriented pipeline for Aurora 0.25° small pretrained (microsoft/aurora, the Earth-system foundation model of Bodnar et al.), pinned to an immutable Hugging Face revision. The repository exposes 6-hourly global weather forecasting from two gridded analyses with autoregressive roll-out, latitude-weighted RMSE per variable and lead time against the persistence baseline, a gridded-window contract with explicit ceilings, a bounded LoRA fine-tuning contract with a portable safetensors adapter, a MODEL_CARD.md at DIMER Model Card Specification 1.1, and a standalone E2E tutorial at DIMER Notebook Specification 2.0.
microsoft/auroraa96afd7ee6d65e3bd2d476f3be798a25a56f2296aurora-0.25-small-pretrained.ckpt (451,339,106 bytes, SHA-256 f80f78de…) and aurora-0.25-static.pickle (12,459,115 bytes, SHA-256 e382103f…) — both pickles, both converted once to safetensors and never served (see below)microsoft-aurora==2.0.1 + torch==2.14.0 + timm + einops (+ xarray/netCDF4 for NetCDF I/O and numcodecs for the tutorial data) — the model class comes from PyPI, no Hub-hosted code and no served pickleBoth upstream assets are pickles, and neither is served. The checkpoint is a torch archive whose pickle references only collections.OrderedDict and torch's tensor-rebuild helpers; the static file is a plain pickle of three numpy arrays. audit_pickle() lists every global each would import without executing anything and refuses anything outside a per-file allow-list (audit digests pinned); convert_model() unpickles each once through a restricted loader — torch.load(weights_only=True) for the checkpoint, a find_class allow-list of two numpy names for the static fields — applies the upstream compatibility shim, loads AuroraSmallPretrained strictly, and writes aurora-0.25-small-pretrained.safetensors (451,230,408 bytes) and aurora-0.25-static.safetensors (12,459,136 bytes), whose digests are pinned in pipeline.py; from_pretrained() loads only those. The conversion is deterministic and its fidelity is measured against upstream's own regression fixture: relative mean deviations 1.2×10⁻⁷ to 3.5×10⁻⁵ per variable, within the upstream tolerances (docs/WEIGHTS.md).
The tutorial runs the model six times coarser than it was trained. The data is real ERA5 regridded to 1.5° by WeatherBench 2 (four two-day windows, 57 digest-pinned Zarr objects, about 196 MB, no credential), because a native 0.25° window is sixty times larger. That makes the frozen model's numbers an out-of-distribution measurement — it loses to persistence on eight of nine variables at 6 h — and makes the adaptation contract the point: a 540 k-parameter LoRA fine-tuning on twelve one-step forecasts brings the mean skill below persistence at every lead to 24 h on a held-out window from a different year. None of it says anything about Aurora at its native resolution.
from aurora_earth_system_pipeline import AuroraPipeline, fetch_sample_dataset, persistence_only
pipe = AuroraPipeline.from_pretrained(use_lora=True) # verifies the snapshot, audits + converts both pickles once, loads safetensors
windows = fetch_sample_dataset() # four 8-step windows of ERA5 at 1.5° from pinned WeatherBench 2 objects
test = windows["test-2021-10"]
forecast = pipe.predict(test, origin=1, steps=4) # 6 h .. 24 h from the analysis at step 1
print(forecast["forecasts"][0]["valid_time"], forecast["forecasts"][0]["surf"]["2t"].shape)
print(persistence_only(test, max_lead_steps=4)["variables"]["z500"])
print(pipe.evaluate(test, max_lead_steps=4)["summary"]) # frozen model vs persistence, per lead
pipe.adapt([windows["train-2019-01"], windows["train-2019-07"]], windows["val-2020-04"])
print(pipe.evaluate(test, max_lead_steps=4)["summary"]) # adapted model
pipe.save_artifact("outputs/adapter")
predict(), evaluate() and adapt() take windows — {lat, lon, levels, times, surf, atmos, static} mappings on a global equiangular grid (17..721 latitudes spanning the poles, a multiple of 4 or one more; 32..1440 longitudes covering the full circle, a multiple of 4), exactly the 13 standard pressure levels, 3..64 analyses spaced exactly 6 h apart, SI units within plausibility ranges. The model predicts the rows the patch size covers (121 → 120 at 1.5°). Validation is structural: nothing checks that a window is dynamically consistent or real.
weights/aurora-0.25-small/ README.md aurora-0.25-small-pretrained.ckpt aurora-0.25-static.pickle dimer-base-manifest.json
aurora-0.25-small-pretrained.safetensors aurora-0.25-static.safetensors (git-ignored, converted)
weights/wb2-era5-1p5deg/ the 57 pinned WeatherBench 2 objects, cached on first fetch (git-ignored)
from_pretrained() calls stage_missing_files() (fetches only absent manifest entries, only at the pinned revision, only with allow_download=True) then verify_snapshot() (byte size + SHA-256 of all 3 manifest entries and of the converted files when present), converts the pickles when the safetensors are absent, and refuses on the first mismatch. With require_source=False digest-verified converted files are accepted without the pickles — the DIMER-hosted shape. docs/WEIGHTS.md records the provenance, the audits, the conversion, the fidelity check and the DIMER hosting notes.
fetch_sample_dataset() assembles four windows — train-2019-01 (2018-12-31 to 2019-01-01), train-2019-07, val-2020-04, test-2021-10 — of eight 6-hourly ERA5 analyses at 1.5° (240 × 121, with poles) from WeatherBench 2's public bucket: every Zarr chunk, .zarray descriptor and coordinate is pinned by byte size and SHA-256 in WB2_OBJECTS, refused on mismatch before decoding, decoded with numcodecs (Blosc/LZ4) and reordered into the Aurora layout. Distinct years and seasons per role, so nothing in the test window is a near-duplicate of anything trained on. ERA5 is © ECMWF / Copernicus Climate Change Service (licence to use Copernicus products). write_window_netcdf / load_byod_dataset round-trip NetCDF, the BYOD format.
save_artifact(dir) writes adapter.safetensors (the trained tensors — the 80 LoRA tensors, about 2 MB; plus the embeddings and heads with trainable="lora+heads") and a manifest.json recording the artifact format, the exact base model id and revision, the converted-base digests, the tensor names, the file size and SHA-256, the training configuration and the epoch history. AuroraPipeline.from_artifact(dir) re-verifies the base files, checks the manifest and digest before deserialising, rebuilds the model with LoRA and overlays the tensors.
pip install -e . --no-deps
pytest -q -o addopts= tests
Tests are offline: crafted pickles, temporary manifests, synthetic windows and an injected fetcher, never the weights; tests/test_model_backed.py runs only where microsoft-aurora and the converted files are present (strict load, deterministic cropped forecasts, LoRA neutrality, a one-epoch adaptation and artifact round trip) and is skipped in CI.
tutorials/aurora_earth_system_colab.ipynb is declared E2E and is standalone (DIMER Notebook Specification 2.2 §4): it is generated by tools/build_notebook.py (/3) from tools/notebook_template.py and carries the package's 4 modules, the stage runner tools/tutorial_stages.py, the hash-locked tutorials/requirements-colab.lock.txt, the snapshot manifest and the licence, each verified by SHA-256, so the exported .ipynb keeps working without this repository being reachable. Nothing is installed into the notebook kernel: a pinned uv builds an isolated CPython 3.12.12 environment from the lock with --require-hashes, and every stage runs there in its own process (no restart is needed; Linux x86_64 runtimes only). It downloads, audits and converts the pinned pickles, fetches the pinned data, and runs the sample path: validation against the schema and the configured workflow, a 24-hour roll-out, persistence and frozen-model baselines per lead, LoRA fine-tuning, held-out evaluation with the adapter rebuilt from its files, a NetCDF forecast from a new origin, reload parity in a fresh process, and an optional activity. The guided layer (how to use, roadmap, glossary, predictions, checkpoints, troubleshooting, conclusion template) is part of the notebook. Do not edit the notebook by hand; regenerate it (python tools/build_notebook.py; --check is enforced by the validator and CI).
Candidate — the notebook was regenerated in the isolated-environment pattern to fix review findings AUR-M1..M4 / AUR-m1..m6; a hosted Colab T4 default-path run of the new blob (a59bed4, blob cfa631b6) completed in one pass with no restart and 0 errors on 2026-10-03, while the activity, BYOD and Kaggle runs are not recorded yet. The 2026-09-19 Kaggle T4 run of the previous blob 42fb3883 needed a restart after the install cell (2 passes), which is not a one-pass Run all; see docs/release-verification.md and STATUS.md. Static and unit checks — including the standalone generator parity checks — are necessary but never the evidence; the recorded hosted runs are.
microsoft/aurora; the microsoft/aurora code is MIT), staged from the pinned revision and converted, not modified, into the served safetensors.LICENSE).This repository’s code and accompanying documentation were developed with generative AI assistance for code development and technical writing under maintainer direction. The maintainer remains responsible for reviewing the implementation, validating results, and making release decisions. AI assistance does not constitute independent verification, provider endorsement, or release approval.
DIMER pipeline for Aurora 0.25° small: audited pickle conversion to safetensors, 6-hourly global forecasting against persistence, and bounded LoRA fine-tuning on pinned ERA5 windows
Jupyter Notebook
0
50 commits
updated Oct 14, 2026
DIMER-oriented pipeline for Aurora 0.25° small pretrained (microsoft/aurora, the Earth-system foundation model of Bodnar et al.), pinned to an immutable Hugging Face revision. The repository exposes 6-hourly global weather forecasting from two gridded analyses with autoregressive roll-out, latitude-weighted RMSE per variable and lead time against the persistence baseline, a gridded-window contract with explicit ceilings, a bounded LoRA fine-tuning contract with a portable safetensors adapter, a MODEL_CARD.md at DIMER Model Card Specification 1.1, and a standalone E2E tutorial at DIMER Notebook Specification 2.0.
microsoft/auroraa96afd7ee6d65e3bd2d476f3be798a25a56f2296aurora-0.25-small-pretrained.ckpt (451,339,106 bytes, SHA-256 f80f78de…) and aurora-0.25-static.pickle (12,459,115 bytes, SHA-256 e382103f…) — both pickles, both converted once to safetensors and never served (see below)microsoft-aurora==2.0.1 + torch==2.14.0 + timm + einops (+ xarray/netCDF4 for NetCDF I/O and numcodecs for the tutorial data) — the model class comes from PyPI, no Hub-hosted code and no served pickleBoth upstream assets are pickles, and neither is served. The checkpoint is a torch archive whose pickle references only collections.OrderedDict and torch's tensor-rebuild helpers; the static file is a plain pickle of three numpy arrays. audit_pickle() lists every global each would import without executing anything and refuses anything outside a per-file allow-list (audit digests pinned); convert_model() unpickles each once through a restricted loader — torch.load(weights_only=True) for the checkpoint, a find_class allow-list of two numpy names for the static fields — applies the upstream compatibility shim, loads AuroraSmallPretrained strictly, and writes aurora-0.25-small-pretrained.safetensors (451,230,408 bytes) and aurora-0.25-static.safetensors (12,459,136 bytes), whose digests are pinned in pipeline.py; from_pretrained() loads only those. The conversion is deterministic and its fidelity is measured against upstream's own regression fixture: relative mean deviations 1.2×10⁻⁷ to 3.5×10⁻⁵ per variable, within the upstream tolerances (docs/WEIGHTS.md).
The tutorial runs the model six times coarser than it was trained. The data is real ERA5 regridded to 1.5° by WeatherBench 2 (four two-day windows, 57 digest-pinned Zarr objects, about 196 MB, no credential), because a native 0.25° window is sixty times larger. That makes the frozen model's numbers an out-of-distribution measurement — it loses to persistence on eight of nine variables at 6 h — and makes the adaptation contract the point: a 540 k-parameter LoRA fine-tuning on twelve one-step forecasts brings the mean skill below persistence at every lead to 24 h on a held-out window from a different year. None of it says anything about Aurora at its native resolution.
from aurora_earth_system_pipeline import AuroraPipeline, fetch_sample_dataset, persistence_only
pipe = AuroraPipeline.from_pretrained(use_lora=True) # verifies the snapshot, audits + converts both pickles once, loads safetensors
windows = fetch_sample_dataset() # four 8-step windows of ERA5 at 1.5° from pinned WeatherBench 2 objects
test = windows["test-2021-10"]
forecast = pipe.predict(test, origin=1, steps=4) # 6 h .. 24 h from the analysis at step 1
print(forecast["forecasts"][0]["valid_time"], forecast["forecasts"][0]["surf"]["2t"].shape)
print(persistence_only(test, max_lead_steps=4)["variables"]["z500"])
print(pipe.evaluate(test, max_lead_steps=4)["summary"]) # frozen model vs persistence, per lead
pipe.adapt([windows["train-2019-01"], windows["train-2019-07"]], windows["val-2020-04"])
print(pipe.evaluate(test, max_lead_steps=4)["summary"]) # adapted model
pipe.save_artifact("outputs/adapter")
predict(), evaluate() and adapt() take windows — {lat, lon, levels, times, surf, atmos, static} mappings on a global equiangular grid (17..721 latitudes spanning the poles, a multiple of 4 or one more; 32..1440 longitudes covering the full circle, a multiple of 4), exactly the 13 standard pressure levels, 3..64 analyses spaced exactly 6 h apart, SI units within plausibility ranges. The model predicts the rows the patch size covers (121 → 120 at 1.5°). Validation is structural: nothing checks that a window is dynamically consistent or real.
weights/aurora-0.25-small/ README.md aurora-0.25-small-pretrained.ckpt aurora-0.25-static.pickle dimer-base-manifest.json
aurora-0.25-small-pretrained.safetensors aurora-0.25-static.safetensors (git-ignored, converted)
weights/wb2-era5-1p5deg/ the 57 pinned WeatherBench 2 objects, cached on first fetch (git-ignored)
from_pretrained() calls stage_missing_files() (fetches only absent manifest entries, only at the pinned revision, only with allow_download=True) then verify_snapshot() (byte size + SHA-256 of all 3 manifest entries and of the converted files when present), converts the pickles when the safetensors are absent, and refuses on the first mismatch. With require_source=False digest-verified converted files are accepted without the pickles — the DIMER-hosted shape. docs/WEIGHTS.md records the provenance, the audits, the conversion, the fidelity check and the DIMER hosting notes.
fetch_sample_dataset() assembles four windows — train-2019-01 (2018-12-31 to 2019-01-01), train-2019-07, val-2020-04, test-2021-10 — of eight 6-hourly ERA5 analyses at 1.5° (240 × 121, with poles) from WeatherBench 2's public bucket: every Zarr chunk, .zarray descriptor and coordinate is pinned by byte size and SHA-256 in WB2_OBJECTS, refused on mismatch before decoding, decoded with numcodecs (Blosc/LZ4) and reordered into the Aurora layout. Distinct years and seasons per role, so nothing in the test window is a near-duplicate of anything trained on. ERA5 is © ECMWF / Copernicus Climate Change Service (licence to use Copernicus products). write_window_netcdf / load_byod_dataset round-trip NetCDF, the BYOD format.
save_artifact(dir) writes adapter.safetensors (the trained tensors — the 80 LoRA tensors, about 2 MB; plus the embeddings and heads with trainable="lora+heads") and a manifest.json recording the artifact format, the exact base model id and revision, the converted-base digests, the tensor names, the file size and SHA-256, the training configuration and the epoch history. AuroraPipeline.from_artifact(dir) re-verifies the base files, checks the manifest and digest before deserialising, rebuilds the model with LoRA and overlays the tensors.
pip install -e . --no-deps
pytest -q -o addopts= tests
Tests are offline: crafted pickles, temporary manifests, synthetic windows and an injected fetcher, never the weights; tests/test_model_backed.py runs only where microsoft-aurora and the converted files are present (strict load, deterministic cropped forecasts, LoRA neutrality, a one-epoch adaptation and artifact round trip) and is skipped in CI.
tutorials/aurora_earth_system_colab.ipynb is declared E2E and is standalone (DIMER Notebook Specification 2.2 §4): it is generated by tools/build_notebook.py (/3) from tools/notebook_template.py and carries the package's 4 modules, the stage runner tools/tutorial_stages.py, the hash-locked tutorials/requirements-colab.lock.txt, the snapshot manifest and the licence, each verified by SHA-256, so the exported .ipynb keeps working without this repository being reachable. Nothing is installed into the notebook kernel: a pinned uv builds an isolated CPython 3.12.12 environment from the lock with --require-hashes, and every stage runs there in its own process (no restart is needed; Linux x86_64 runtimes only). It downloads, audits and converts the pinned pickles, fetches the pinned data, and runs the sample path: validation against the schema and the configured workflow, a 24-hour roll-out, persistence and frozen-model baselines per lead, LoRA fine-tuning, held-out evaluation with the adapter rebuilt from its files, a NetCDF forecast from a new origin, reload parity in a fresh process, and an optional activity. The guided layer (how to use, roadmap, glossary, predictions, checkpoints, troubleshooting, conclusion template) is part of the notebook. Do not edit the notebook by hand; regenerate it (python tools/build_notebook.py; --check is enforced by the validator and CI).
Candidate — the notebook was regenerated in the isolated-environment pattern to fix review findings AUR-M1..M4 / AUR-m1..m6; a hosted Colab T4 default-path run of the new blob (a59bed4, blob cfa631b6) completed in one pass with no restart and 0 errors on 2026-10-03, while the activity, BYOD and Kaggle runs are not recorded yet. The 2026-09-19 Kaggle T4 run of the previous blob 42fb3883 needed a restart after the install cell (2 passes), which is not a one-pass Run all; see docs/release-verification.md and STATUS.md. Static and unit checks — including the standalone generator parity checks — are necessary but never the evidence; the recorded hosted runs are.
microsoft/aurora; the microsoft/aurora code is MIT), staged from the pinned revision and converted, not modified, into the served safetensors.LICENSE).This repository’s code and accompanying documentation were developed with generative AI assistance for code development and technical writing under maintainer direction. The maintainer remains responsible for reviewing the implementation, validating results, and making release decisions. AI assistance does not constitute independent verification, provider endorsement, or release approval.