tsfm-bench — Inference-cost benchmarking for Time Series Foundation Models

A standardized harness that measures the inference cost — latency, throughput, peak GPU memory, and context-length/batch scaling — of time series foundation models (TSFMs) under matched conditions. Accuracy leaderboards dominate the TSFM literature; this project isolates and quantifies the inference-cost drivers those papers omit.

Models: Chronos, TimesFM, Sundial. Cost metrics use synthetic input (only tensor shape drives latency/throughput/ memory), which removes any dataset dependency and cleanly isolates cost drivers. An optional accuracy sanity check (MASE / sMAPE) runs on a small staged real dataset to confirm the inference setup is faithful.

Environment

• Runtime: the uv venv is seeded from the CUDA-enabled py311 conda env with --system-site-packages so the working torch 2.7+cu126 / CUDA stack is inherited rather than reinstalled.
• GPU: single NVIDIA Tesla T4 (15 GB). The default grid is sized accordingly and the runner records out-of-memory configs as oom instead of crashing.

Setup

# from the repo root (tsfm/)
uv venv --python /opt/conda/envs/py311/bin/python --system-site-packages .venv
source .venv/bin/activate

# runtime deps that are safe to add on top of the inherited stack
uv pip install chronos-forecasting gluonts utilsforecast polars pyyaml tqdm
# dev tooling
uv pip install ruff mypy pytest pytest-mock
# jupyter (kept out of project deps by design)
uv pip install ipykernel ipywidgets

# install this package WITHOUT touching the torch stack
uv pip install -e . --no-deps

# sanity: CUDA must still be available
python -c "import torch; print(torch.cuda.is_available())"   # -> True

Important: always install extra packages that depend on torch/transformers with --no-deps (or uninstall any torch, transformers, huggingface-hub, accelerate that get pulled into the venv). uv installs dependencies into the venv even when they already exist in system site-packages, and the newest torch build requires a CUDA driver newer than this host's (12.2), which silently disables the GPU.

Staging model weights (required for live runs)

The Hub is blocked, so download these on a machine with internet and copy them into the paths below (all are .gitignored). Then run with local_files_only loading, which the adapters use automatically.

For each HuggingFace snapshot, copy the full folder contents (config, tokenizer, *.safetensors). TimesFM may need its extra deps:

uv pip install timesfm --no-deps        # then add any missing pure-python deps it imports

(Optional, accuracy check only) stage a small Monash/M4 subset under data/.

Running

# full sweep from the default config
tsfm-bench --config configs/default.yaml --output-name t4_run

# a subset of models
tsfm-bench --config configs/all_models_bench.yaml --only Chronos TimesFM

# validate the whole pipeline with the weightless dummy model (no weights needed)
python -m tsfm_bench.cli --config configs/smoke.yaml --output-name smoke

Results are written as a tidy Parquet table to results/.

Analysis

After benchmarking, analyze the results with:

# Comprehensive analysis: latency, throughput, memory, quality metrics
python analyze_benchmark.py

# Or load and visualize in notebook
# Open notebooks/demo.ipynb to render comparative latency / throughput / memory / scaling charts

The analysis script computes summary statistics by model, performance across context lengths and batch sizes, memory scaling, and quality metrics (variance stability, warm-up anomalies).

Results schema

One row per (model, context_length, batch_size, horizon) with comprehensive inference-cost metrics:

Status & Metadata

• status: ok / oom / unsupported / skipped
• model, model_key, device, context_length, batch_size, horizon

Latency Metrics (milliseconds)

• latency_ms_mean, std, p50, p95, p99, p999, max
• latency_ms_first_run — warmup-phase latency; compared to mean to detect warm-up effects
• latency_cv — coefficient of variation; flags < 10% for proposal target

Throughput

• throughput_series_per_s — batch_size / (mean_latency_sec)

Memory Breakdown (GiB)

• memory_initial_gib — allocated before inference
• memory_activation_gib — peak - initial (transient tensor memory)
• memory_peak_gib, memory_mean_gib — total allocated
• memory_per_batch_item_mib — peak memory per series for linear scaling analysis

GPU Utilization (when available)

• gpu_util_pct — GPU core utilization % (requires nvidia-ml-py)
• gpu_mem_util_pct — GPU memory utilization %
• gpu_power_watts — power draw (when available)

Quality Checks

• variance_ok — run-to-run CV ≤ 10% threshold
• first_run_anomaly — true if first run > 50% slower than mean (indicates warm-up overhead)

Development

ruff check . && ruff format --check .
mypy src
pytest

Layout

src/tsfm_bench/
  config.py                   # RunConfig / BenchGrid / ModelSpec + YAML loading
  registry.py                 # model key -> adapter factory (lazy imports)
  adapters/                   # base ABC + model adapters
    base.py                   # ForecastAdapter abstract base
    dummy.py                  # Weightless dummy model (testing)
    chronos.py                # Chronos adapter
    timesfm.py                # TimesFM adapter
    sundial_sktime.py         # Sundial adapter (via sktime wrapper)
  data/synthetic.py           # deterministic synthetic context batches
  bench/                      # timing (CUDA events), memory, OOM-safe grid runner
  metrics/accuracy.py         # MASE / sMAPE
  cli.py                      # `tsfm-bench` entry point
configs/                      # YAML configs: default.yaml, all_models_bench.yaml, etc.
notebooks/demo.ipynb          # charts from results Parquet
analyze_benchmark.py          # analysis script: latency, throughput, memory, quality
tests/                        # metrics, timing, synthetic, registry, config