Foundation Model Break-Even Analysis

When Do Foundation Models Pay Off? Data Requirements for Pretrained Time Series Forecasters

Code for measuring break-even points — the training data size at which foundation models (Chronos, Moirai, Lag-Llama) start outperforming classical methods (Naive, ETS, ARIMA, XGBoost) across 30 benchmark datasets.

Quick Start

# Setup
git clone https://github.com/nicolaisi/fm-breakeven.git
cd fm-breakeven
pip install -r requirements.txt

# 0. Pre-fetch all 30 datasets (recommended before long runs)
./run_all.sh download

# 1. Run everything (classical + foundation + Lag-Llama, all 30 datasets)
./run_all.sh

# 2. Or run selectively
./run_all.sh classical      # CPU only, all 30 datasets
./run_all.sh foundation     # GPU: Chronos-Bolt + Moirai
./run_all.sh lagllama       # GPU: Lag-Llama zero-shot only
./run_all.sh new            # New 20 datasets only (classical + foundation)
./run_all.sh rq3            # RQ3 LOO analysis (after experiments complete)

Foundation models require a GPU:

pip install -r requirements-gpu.txt
./run_all.sh foundation

Datasets

30 datasets total: 10 original + 20 new.

Original 10

New 20

All datasets download automatically on first use. Run ./run_all.sh download to pre-fetch.

Models

Classical (CPU):

• Naive (Seasonal Naive)
• ETS (AutoETS via statsforecast)
• ARIMA (AutoARIMA via statsforecast)
• XGBoost (with lag features)

Foundation (GPU):

• Chronos-Bolt S/B (Amazon) — zero-shot, fine-tuned-full, fine-tuned-LoRA
• Moirai S/B/L (Salesforce) — zero-shot, fine-tuned-full, fine-tuned-LoRA
• Lag-Llama — zero-shot only

Experiment Configuration

# Training fractions (% of full training set)
TRAIN_FRACTIONS = [0.02, 0.05, 0.10, 0.20, 0.50, 1.00]

# Random seeds for variance estimation
SEEDS = [0, 1, 2]

# Split: 60% train / 20% val / 20% test (chronological)

Output Files

results/
├── classical_results_*.csv     # Raw classical results
├── foundation_raw_*.csv        # Raw foundation results
├── aggregated_results.csv      # Mean ± std across seeds
├── fitted_parameters.csv       # Power-law fit: Error(n) = A·n^(-α) + B
├── dataset_features.csv        # Features: τ_acf, Hurst, noise ratio, seasonal strength, spectral entropy
└── breakeven_analysis.csv      # Break-even points n*

figures/
├── fig1_scaling_curves.*       # RQ1: scaling curves per dataset
├── fig2_feature_space.*        # RQ2: dataset feature embedding
├── fig3_finetuning.*           # RQ3: fine-tuning vs. zero-shot
└── fig4_pareto.*               # RQ4: accuracy–cost Pareto frontier

logs/
└── *_<timestamp>.log           # Per-run logs

Repository Structure

fm-breakeven/
├── run_all.sh                        # Main entry point
├── audit_paper.py                    # Numerical audit of tex against ground-truth CSVs
├── rq3_analysis.py                   # RQ3 decision-tree analysis
├── plot_fig1.py                      # Figure 1: scaling curves
├── plot_fig2.py                      # Figure 2: feature space
├── plot_fig3.py                      # Figure 3: fine-tuning
├── src/
│   ├── data_loader.py                # Dataset loading (auto-download, 30 datasets)
│   ├── models.py                     # Classical + foundation model wrappers
│   ├── config.py                     # Experiment configuration
│   ├── features.py                   # Feature extraction (τ_acf, H, S, Φ)
│   ├── analysis.py                   # Scaling curve fitting, break-even computation
│   ├── experiments.py                # Experiment runner
│   └── visualization.py             # Plotting utilities
├── scripts/
│   ├── run_classical.py              # Run classical experiments (CPU)
│   ├── run_foundation.py             # Run foundation model experiments (GPU)
│   ├── run_quick.py                  # Minimal run for installation verification
│   ├── smoke_test.py                 # Sanity check all 30 datasets load and run
│   ├── analyze_results.py            # Aggregate results and generate figures
│   ├── analyze_finetuning.py         # Fine-tuning analysis: full vs LoRA vs zero-shot
│   ├── validate_findings.py          # Validates all 5 paper findings against raw CSVs
│   ├── compute_features.py           # Dataset feature extraction script
│   ├── compute_costs.py              # Extract per-model GPU-seconds; feeds plot_fig4.py
│   ├── generate_table5.py            # Generate Table V (computational cost) as LaTeX
│   ├── plot_fig4.py                  # Figure 4: Pareto frontier
│   ├── download_datasets.py          # Pre-fetch all 30 datasets
│   └── rq3_analysis_extended.py      # RQ3 LOO validation
├── requirements.txt                  # CPU dependencies
└── requirements-gpu.txt              # GPU dependencies (Chronos, Moirai, Lag-Llama)

Reproducing Paper Findings

Run scripts/validate_findings.py to verify every number cited in Findings 1–5 against the raw results CSVs:

python scripts/validate_findings.py

Runs 25 checks covering all 5 findings. Exits with code 0 if all pass, 1 if any fail. Example output:

FINDING 1: Zero-shot is already competitive
  ✓  FM zero-shot beats fully-trained classical on 19/30 datasets
  ✓  15 datasets are FM-dominant (classical never beats FM at any fraction)
...
ALL CHECKS PASSED (25 checks)

Fine-Tuning Analysis

Run scripts/analyze_finetuning.py to reproduce the numbers cited in Finding 3 of the paper:

python scripts/analyze_finetuning.py

This script produces four sections:

Key findings from the analysis:

• Full fine-tuning beats LoRA on 22/30 datasets and zero-shot on 21/30 at full training data
• At low fractions (≤5%), 109 catastrophic cases (MASE > 10) across Moirai and Chronos
• Worst single-model inflation: Moirai-S on Solar-10min (MASE 23.6 vs zero-shot 0.92)
• LoRA has lower seed std (0.095 vs 0.129 at frac≤5%) — more stable in the low-data regime

Key Results

All numbers below are verified by scripts/validate_findings.py (25 automated checks).

# Reproduce all findings in one command
python scripts/validate_findings.py
# Expected output: ALL CHECKS PASSED (25 checks)

GPU Requirements

Foundation models need:

• CUDA-enabled GPU (16+ GB VRAM recommended)
• ~8 GPU-hours for full experiment suite (Chronos/Moirai ≈7.5h, Lag-Llama ≈25min on RTX 5090)

Per-Model Runtime (RTX 5090, mean per dataset-fraction-seed)

Runtimes measured from duration_s column in raw results CSVs. Full fine-tuning adds ~50% over LoRA.

VRAM Requirements

Consumer GPUs (RTX 4090/5090 with 24-32GB) can run everything, including fine-tuning.

Suggested Workflow

# 1. Test single model + dataset first (~10 min)
python scripts/run_foundation.py --gpu \
    --datasets ETTh1 \
    --models Chronos-Bolt-S

# 2. Expand to more models
python scripts/run_foundation.py --gpu \
    --datasets ETTh1 Exchange \
    --models Chronos-Bolt-S Chronos-Bolt-B Moirai-S

# 3. Full run overnight (~8-12 hours on RTX 5090)
./run_all.sh

Only use a compute cluster if you need results faster. RTX 5090 ≈ 80% A100 speed for this workload.

Citation

@inproceedings{jerome2026breakeven,
  title={When Do Foundation Models Pay Off?
         Data Requirements for Pretrained Time Series Forecasters},
  author={Nicholas Tan Jerome and Frank Simon},
  booktitle={IEEE International Conference on Data Mining (ICDM)},
  year={2026}
}

License

MIT