TL;DR — Clean-data leaderboards do not predict robustness. TS-Fault stops treating faults as i.i.d. noise. It injects four structurally-coupled failure modes — abstracted from the anatomy of real grid-scale failures — into the input window of nine standard forecasting datasets across six domains, then measures how far each model degrades. Under the two mechanism-level modes, the ranking is reordered: state-of-the-art Transformers and foundation models collapse, while a plain LSTM/GRU — or even a naïve baseline — becomes the most robust.

The TS-Fault pipeline. A clean pair (X, Y) is turned into a structured instance: a window-importance score selects the prediction-critical window W★, a parameterized fault operator TΘ acts there at a chosen severity, and the scenario parameters Θ and difficulty δ are reported with the instance and used to condition evaluation.

📌 Why TS-Fault?

For two decades, long-horizon forecasting progress has been measured by one number — average error on clean, complete, evenly-sampled held-out data — under the implicit assumption that it predicts deployed reliability. Real faults are not white noise. They are structured events: a transient with onset–peak–decay, a silently broken cross-variable dependency, a regime change coupled with block-missingness, a fault cascading through a sensing pipeline. None of these can be expressed by i.i.d. perturbations, no matter how the variance or masking rate is tuned.

TS-Fault changes the object of evaluation, replacing the clean test pair (X, Y) with a structured instance (X̃, Ỹ, Θ, δ) produced by an explicit, parameterized fault operator. Because Θ is exposed, a model's degradation can be attributed to a named mechanism at a tunable severity — turning a pass/fail noise test into an ablation-style diagnostic tool.

Headline findings

🧩 The four failure modes

Each mode corrupts only the lookback window (L = 336); the forecast target (H = 96) is never touched, so degradation isolates the cost of a corrupted history. A single difficulty scalar d ∈ {0.2, 0.4, 0.6, 0.8, 1.0} (written d02 … d10) sweeps severity monotonically. The modes span a 2 × 2 taxonomy — observation- vs. mechanism-level × univariate vs. multivariate — and form a constructive minimum: omitting any one renders an entire class of deployment failure invisible.

Observation-level (I, II) corrupt observations while the data-generating process stays intact → mild. Mechanism-level (III, IV) rewrite the process itself → severe, and account for 100% of catastrophic failures.

📂 Repository contents

This repository ships the data-generation pipeline, model implementations, evaluation drivers, trained checkpoints, and the complete results workbook. The four fault generators are included in full, so the perturbed dataset can be regenerated from scratch.

.
├── README.md
├── requirements.txt
│
├── benchmark.py                 # benchmark orchestration helpers
├── dataset_loader.py            # load + normalise the clean CSVs into windows
├── window_selector.py           # select the anchor lookback / horizon windows
│
├── Mode1.py                     # Fault Mode I   — Time-Warped Shock
├── Mode2.py                     # Fault Mode II  — Dependency-Fracture Shock
├── Mode3.py                     # Fault Mode III — Regime-Transition Missingness
├── Mode4.py                     # Fault Mode IV  — Cascading Sensor-to-System Failure
├── run_TS-Fault.py              # ▶ MAIN entry point: build the perturbed benchmark
│
├── classical_models.py          # statistical / linear / recurrent / conv. forecasters
├── foundation_models.py         # TimesFM / Chronos / Moirai (zero-shot wrappers)
├── eval_classical_phase1.py     # evaluate classical + baseline models → results CSV
├── eval_foundation_phase1.py    # evaluate foundation models → results CSV
├── checkpoints/                 # released trained weights
├── ablation_placement.py        # window-placement ablation: top-K S(W) vs random vs anti-selected
├── ablation_lambda.py           # one-at-a-time sensitivity of the S(W) weights λ
├── adapt_foundation.py          # LP / FT / FT-fault adaptation of the three foundation models
├── compose_faults.py            # ⊛ compound faults: T_B ∘ T_A at matched severity
└── figures/                     # paper figures used in this README

The 21 evaluated forecasters

The paper evaluates 21 models across six methodological families, so the conclusions are not an artifact of a single inductive bias:

Statistical, linear, and recurrent/convolutional models live in classical_models.py; the eight Transformer-family models are trained via the Time-Series-Library (see § Transformer-family); the three foundation models run strictly zero-shot via foundation_models.py. The repository's classical_models.py additionally bundles a few exploratory baselines (e.g. RandomForest, XGBoost) that fall outside the 21-model paper subset.

📊 Datasets (clean originals)

TS-Fault perturbs nine long-horizon datasets spanning six domains — energy, load, climate, finance, IT/Ops, and healthcare — deliberately chosen to cover a wide range of dimensionality (7 → 321 channels) and granularity (5-second → daily). Every clean window is a length-336 history and a length-96 target. We do not redistribute the raw data — download it and place each file at the path the loader expects.

from huggingface_hub import hf_hub_download
for f in ["ETT-small/ETTh1.csv", "ETT-small/ETTh2.csv",
          "ETT-small/ETTm1.csv", "ETT-small/ETTm2.csv",
          "electricity/electricity.csv", "weather/weather.csv",
          "exchange_rate/exchange_rate.csv"]:
    hf_hub_download("thuml/Time-Series-Library", f, repo_type="dataset")

🚀 Quick start

1 · Install

git clone https://github.com/Ray-zyy/TS-Fault.git
cd TS-Fault
pip install -r requirements.txt

2 · Get the clean datasets

Download the nine CSVs (see § Datasets) into:

dataset/ETT-small/{ETTh1,ETTh2,ETTm1,ETTm2}.csv
dataset/electricity/electricity.csv
dataset/weather/weather.csv
dataset/exchange_rate/exchange_rate.csv
dataset/bizitobs/service_kpi.csv
dataset/vitaldb/vitaldb_core.csv  

3 · Generate the perturbed benchmark

run_TS-Fault.py slides the anchor windows (window_selector.py), loads & normalises the clean series (dataset_loader.py), and applies each of the four fault modes (Mode1.py … Mode4.py) across all five difficulties:

python run_TS-Fault.py \
    --data_root ./dataset \
    --out ./TS-Fault_output \
    --n_windows 20

This writes one .npz per (dataset, Mode, difficulty) to TS-Fault_output/<Dataset>/<Dataset>_Mode<k>_d<dd>.npz — 9 datasets × 4 modes × 5 difficulties = 180 files, each containing:

4 · Evaluate

Classical / baseline models (deep ones train once per dataset on clean windows; statistical ones need no training):

python eval_classical_phase1.py \
    --models Naive SeasonalNaive ARIMA ETS \
             DLinear NLinear NBEATS \
             LSTM GRU TCN \
    --npz_root ./TS-Fault_output \
    --out ./results/eval_classical.csv --gpu 0 --resume

Foundation models (zero-shot — see § Foundation models):

python eval_foundation_phase1.py \
    --models timesfm chronos moirai \
    --npz_root ./TS-Fault_output \
    --out ./results/eval_foundation.csv --gpu 0 --resume

Each evaluator emits the canonical wide schema, one row per (model, dataset, Mode, difficulty):

model, dataset, Mode, difficulty, mse_corrupt, mae_corrupt, mse_clean, mae_clean, n_samples, time_sec

mse_clean is the error on the clean input window, mse_corrupt on the perturbed window — both predicting the same untouched target. The robustness ratio r = mse_corrupt / mse_clean isolates the cost of corrupted history; r = 1 is perfect robustness and r ≥ 10 is a catastrophic failure. Concatenate the per-group CSVs (classical + foundation + the TSLib Transformer rows) into the master workbook eval_results_full.xlsx (also mirrored as the flat eval_results_full_23.csv).

🧪 Evaluating the Transformer-family models

The eight Transformer/attention models (PatchTST, iTransformer, Autoformer, FEDformer, Nonstationary-Transformer, TimeMixer, TimeXer, TimesNet) are trained and run with the Time-Series-Library (TSLib).

1. Clone TSLib and place the same nine clean CSVs under its ./dataset/.

2. Train / evaluate each model with run.py under the TS-Fault protocol (seq_len=336, pred_len=96, features=M, MSE loss). Example:

   python -u run.py \
       --task_name long_term_forecast --is_training 1 \
       --root_path ./dataset/ETT-small/ --data_path ETTh1.csv \
       --model_id ETTh1_336_96 --model PatchTST --data ETTh1 \
       --features M --seq_len 336 --label_len 48 --pred_len 96 \
       --e_layers 2 --d_layers 1 --enc_in 7 --dec_in 7 --c_out 7 \
       --train_epochs 10 --batch_size 32 --learning_rate 1e-4 --itr 1
   

3. Score each trained checkpoint on the TS-Fault .npz windows (clean vs. corrupt) and write rows in the same wide schema, then merge into the master workbook eval_results_full.xlsx.

🧱 Installing the foundation models

These libraries pull heavy / conflicting dependencies. The recipe that works (verified on CUDA 12.1, torch 2.5.1):

# TimesFM + Chronos are clean pip installs
pip install timesfm chronos-forecasting

# Moirai (uni2ts) pins torch<2.5 and WILL try to downgrade torch.
# Install WITHOUT deps so it cannot touch torch, then hand-add what it needs:
git clone https://github.com/SalesforceAIResearch/uni2ts.git
cd uni2ts && pip install -e . --no-deps && touch .env
pip install "gluonts==0.14.4" "einops==0.7.0" jaxtyping --no-deps

Behind a mirror, set export HF_ENDPOINT=https://hf-mirror.com before the first run. All three run zero-shot (no fine-tuning) and are wrapped channel-independently for multivariate inputs.

🏆 Results

The complete results workbook

Every number in this README — and a great many that did not fit — is shipped in eval_results_full.xlsx, the full evaluation table. It holds one row per (model, dataset, mode, difficulty) cell for the entire grid of 21 models × 9 datasets × 4 modes × 5 severities ≈ 3,800 rows (ARIMA is omitted on the 321-channel Electricity, where per-series fitting is impractical), under the same wide schema produced by the evaluators:

model · dataset · Mode · difficulty · mse_corrupt · mae_corrupt · mse_clean · mae_clean · n_samples · time_sec

From these columns the robustness ratio r = mse_corrupt / mse_clean, the relative degradation (r − 1) × 100%, the severity slope d10/d02, and every per-mode / per-dataset aggregate in the paper can be recomputed directly — no rerun required. A flat-CSV mirror is provided as eval_results_full_23.csv.

Aggregate — the clean and faulted leaderboards are near mirror images

The most robust models (GRU, LSTM, TCN, near-unit ratios) are only mid-pack on clean accuracy, while the clean-accuracy leaders (N-BEATS, TimesFM, iTransformer, TimeXer) sit at the bottom of the robustness order with ratios in the hundreds.

Left: clean/faulted MSE-MAE and robustness ratio. Center: degradation across severities d02→d10 and the d10/d02 sensitivity slope. Right: per-mode relative degradation — note the jump from single-digit % under Modes I/II to thousands–to–tens-of-thousands % under Modes III/IV.

Rank reordering — preserved under observation-level faults, destroyed under mechanism-level

Per-dataset — fragility is structural; dimensionality sets the amplitude

The model ordering is highly consistent across datasets (fragility is a property of the architecture), but dense cross-channel correlation amplifies it: on the 321-channel Electricity, TimesFM's ratio peaks at 2090 and TimeXer at 1112, while GRU/LSTM stay near 1.2 everywhere.

MSE on corrupted inputs per dataset (lower is better). Recurrent models (LSTM, GRU) sit at the bottom of every panel; foundation models (gold) sit at the top.

Catastrophic failures (r ≥ 10): 884 total — 0 in Mode I, 0 in Mode II, 537 in Mode III (85.9% of its cells), 347 in Mode IV (55.5%). Mechanism-level modes account for 100% of them.

Three new domains — finance, IT/Ops, healthcare

The added datasets were chosen to break the energy/climate monoculture of the original release, not to make the benchmark easier. They do not change the verdict.

Which mechanism dominates depends on the domain — and it depends on it in the direction the taxonomy predicts.

(mean r over the 21 models, averaged across the five severities)

🧪 Extended experiments

Four follow-up studies run on top of the same released grid. Each has its own driver script and its own results CSV, so every number below can be recomputed without touching the main pipeline.

1 · Compound faults — do co-occurring failures interact?

Real incidents rarely arrive one at a time. Because the four modes are operators, they compose: TΘ = TΘ_B ∘ TΘ_A (Eq. 19). Both operators act on the same critical window W★, each keeps its own Θ, and at every severity the two constituents are matched at the same nominal difficulty, κ_A(Θ_A) = κ_B(Θ_B) = δ_s. One fixed application order per pair — composition is not commutative, so the order is itself a scenario parameter.

python compose_faults.py \
    --pairs I+II I+III I+IV II+III II+IV III+IV \
    --npz_root ./TS-Fault_output --out ./results_compound.csv

The amplification factor Ψ(A∘B) = r(A∘B) / max(r_A, r_B) asks whether a compound fault is worse than its more damaging half.

• No pair cancels. All six satisfy Ψ ≥ 1.07 — a compound fault is never easier than its worse half.
• Mechanism × mechanism is superadditive. III ∘ IV reaches Ψ = 3.53: a regime switch that a cascade then propagates is more than three times worse than either alone.
• Two observation-level faults do not add up to a mechanism-level one. I ∘ II tops out at r = 1.24. The stratification of Finding 2 survives composition — it is not an artifact of testing one mode at a time.

2 · Where the fault is placed — is S(W) doing any work?

S(W) is the claim that a benchmark should stress the window a model relies on. If placement did not matter, TS-Fault would be a noise test with extra steps. We compare three placement policies at severity s₃ (d06), over 9 datasets × 21 models × 5 seeds, with random placement δ-matched so only the location differs:

python ablation_placement.py --policy topk random anti --severity d06 --seeds 5 \
    --out ./results_placement.csv

Anti-selected < Random < Top-K in every mode, on both metrics. Two consequences beyond "the score works": importance-based placement cuts seed-to-seed variability by 3–4× (the IQR/median column), so a given robustness number means something; and as placement is weakened the faulted ranking drifts back toward the clean ranking (Mode III ρ: 0.023 → 0.147 → 0.286). Random placement would have partially hidden the model-selection risk that motivates the benchmark.

3 · Sensitivity to the S(W) weights

The per-mode weights and window hyperparameters, in full:

One-at-a-time perturbation at severity s₃ (ablation_lambda.py):

The weight each mode is designed around is the one that matters: zeroing λ₄ (occlusion) costs 42–51% of the degradation in Modes I/II/IV, and zeroing λ₁ (change-point) costs 54% in Mode III. Removing a weight the mode does not emphasise moves the result by at most 2.3%. Crucially, the model ranking is stable throughout (ρ ≥ 0.871): the weights control how hard the benchmark is, not who wins.

4 · Can adaptation fix the foundation models?

The zero-shot fragility result invites the obvious question. We adapted all three foundation models under four settings and re-ran the full grid:

python adapt_foundation.py --model timesfm --setting ft_fault \
    --train_severities d02 d04 --eval_severities d06 d08 d10 \
    --out ./results_foundation_adapt.csv

• Clean-data adaptation buys accuracy, not robustness. LP improves TimesFM's ratio only 1.6× (555 → 344.5). Full fine-tuning cuts absolute faulted error 2.6–3.5× yet leaves ratios at 110–188 — still two orders of magnitude from an LSTM.
• Only fault structure in the training signal closes the gap. FT-fault drops the ratios to 12.2 / 14.3 / 9.74 and faulted MSE by a further 7.4–8.6× over clean fine-tuning, at a clean-accuracy cost of just 9–10%. Note this is measured on held-out severities — the models never saw s₃–s₅ during adaptation, so this is generalisation along κ, not memorisation of a severity.
• The gap is narrowed, not closed. The best adapted model (Moirai FT-fault, r = 9.74) is still ~9× less robust than a plain trained LSTM (r = 1.07). The accurate-and-robust quadrant remains empty.

5 · Worst-case risk — average behavior can hide adverse configurations

Average risk (AVG) describes typical behavior over the released fault grid; worst-case risk (WC) asks for the most damaging released configuration:

$$ \mathrm{WC}(f,\mathcal F)=\sup_{\Theta\in\Phi}\mathbb E[\ell\mid\Theta]. $$

We report two complementary quantities:

• WC MSE — the maximum per-configuration faulted MSE over the released grid, directly discretising Eq. (3).
• WC r — within each dataset, the maximum configuration-level robustness ratio, then averaged across the datasets. This uses the same within-dataset-then-average convention as AVG r, so the two columns are directly comparable.

Worst-case risk over all four modes. WC/AVG is the ratio between the worst-case and average robustness risks.

Per-mode worst-case risk. Each cell reports WC r / WC MSE.

Three conclusions follow:

• WC is not a restatement of AVG. The WC/AVG gap is only 1.2–1.3× for LSTM/GRU but roughly 7× for the attention and foundation-model cluster. Typical degradation therefore does not bound adverse degradation.
• The worst configurations are mechanism-level. Twenty of the 21 models attain their overall WC in Mode III and GRU attains it in Mode IV. Twenty peak at d10; TCN is the sole exception, peaking at d08.
• Observation-level faults remain bounded. No model exceeds WC r = 1.64 in Modes I/II, whereas 19 of 21 exceed WC r = 10 in both Modes III and IV. LSTM and GRU are the only exceptions.

🔁 Reproducibility

TS-Fault is designed to be fully reproducible. Because faulted instances are produced by an explicit operator at evaluation time, the benchmark can be regenerated at any severity by re-sweeping κ, and previously-unexposed Θ combinations can be held out at release time to guard against benchmark gaming. We release:

• the parameterized fault generators for all four modes, with their Θ schemas and the unified window-importance front-end — plus compose_faults.py for the compound-fault operator TΘ_B ∘ TΘ_A;
• the exact λ and β values used by S(W) and κ in every mode, with the placement and weight ablations that probe them (ablation_placement.py, ablation_lambda.py);
• the evaluation harness with per-model configurations, and adapt_foundation.py for the LP / FT / FT-fault adaptation protocol;

📜 License & acknowledgements

TS-Fault code is released under the MIT licence. Some baseline implementations are adapted from external sources and retain their own licences: TCN from locuslab/TCN (MIT), NLinear/DLinear from cure-lab/LTSF-Linear (Apache-2.0), and N-BEATS from ServiceNow/N-BEATS (CC-BY-NC-4.0, non-commercial). The eight Transformer baselines come from the Time-Series-Library. Datasets (ETT, Electricity, Weather) are released by their original authors under CC BY 4.0 and are not redistributed here.