ORCA: Blackbox Time-Series Forecasting Refinement Framework

A unified evaluation framework for blackbox time-series forecasting model refinement. This project runs end-to-end experiments on CSV datasets, comparing multiple refiner architectures across frozen foundation models.

Scope:

• CSV datasets only (no external data loaders)
• Frozen base models (no gradient backpropagation to base model)
• Online and batch refiner training modes
• Reproducible summary CSVs with per-dataset, per-model metrics

---

1. Environment Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Key dependencies:

• PyTorch 2.4.0+
• GluonTS (evaluation metrics)
• transformers 4.40.1+ (model backends)
• chronos-forecasting, timesfm, uni2ts (foundation model packages)

---

2. Dataset Preparation

2.1 Supported Datasets

8 benchmark time-series datasets: ETTh1, ETTh2, ETTm1, ETTm2, Exchange, Weather, Traffic, Electricity

2.2 Download Datasets

Download all datasets:

python -m data.download_CSV --dataset all

Download single dataset:

python -m data.download_CSV --dataset ETTh1

Cached files are stored in data/data_cache/. Use --force to re-download.

2.3 Use Custom CSV

Provide any CSV file directly:

python run.py --csv_path /path/to/your/data.csv --target_column all

• --target_column all: multivariate (all numeric columns)
• --target_column OT: single column (or fallback to last column)

---

3. Base Models

3.1 Default Models (run with --model all)

3.2 Additional Models (not in default all)

• moirai-1-small, moirai-1-base, moirai-1-large

To use them, explicitly specify:

python run.py --model moirai-1-large --dataset ETTh1

3.3 Model Checkpoints

Default path: path_to_your_model

Override with --tsfm_model_prefix:

python run.py --tsfm_model_prefix /your/path/to/models

Or specify exact path for single model:

python run.py --model chronos-2 --tsfm_local_path /path/to/chronos-2

---

4. Refiners (Core Components)

9 refiner architectures, categorized into two groups:

4.1 Our Proposed Refiners (Linear/Bay/Attn/Bay_Attn)

Support full ablation study with configurable parameters:

Configurable dimensions:

Input mode semantics (--refiner_input):

4.2 Baseline Refiners (AdaY/DSOF/TAFAS/SOLID/ELF)

External methods integrated for comparison. Limited configurability:

Router mode (--baseline_router):

• Disabled (default): Vanilla baseline refiner
• Enabled: Activates Boltzmann router for dynamic selection between base model and refiner output

---

5. Running Experiments

5.1 Main Experiment (Proposed Method)

Bayesian refiner, OrCA input (xy), bayesian update, buffer=3000, batch=256:

python run.py \
  --dataset all \
  --model all \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 256 \
  --pred_len 96 \
  --context_length 520 \
  --device cuda

This is the core configuration for the proposed method (OrCA).

5.2 Input Ablation Study

Test all 5 input variants:

python run.py \
  --dataset all \
  --model all \
  --refiner bay \
  --refiner_input xy all x y e_past \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 256 \
  --pred_len 96

This runs 5 experiments (one per input mode), results saved separately.

5.3 Update Rule Ablation

Compare plain vs bayesian update:

python run.py \
  --refiner bay \
  --refiner_input xy \
  --update_rule plain bayesian \
  --online_buffer_windows 3000 \
  --batch 256

5.4 Buffer Size Study

python run.py \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --online_buffer_windows 2000 3000 4000 \
  --batch 256

5.5 Batch Size Study (Bay Refiner)

python run.py \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 128 256 512

5.6 Baseline Refiners (with Router)

# Vanilla baseline
python run.py --refiner dsof --dataset ETTh1 --model chronos-2

# With Boltzmann router
python run.py --refiner dsof --dataset ETTh1 --model chronos-2 --baseline_router

5.7 Control Group (No Refiner)

Baseline metrics are automatically computed for every run (before refiner is applied). Results include both baseline and refined rows in output CSVs.

5.8 Multi-Prediction Length

python run.py \
  --pred_len 30 96 336 \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian

Automatically generates per-pred_len CSVs + one pred_avg average file.

5.9 Single Dataset + Single Model

python run.py \
  --dataset ETTh1 \
  --model moirai-2 \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 256 \
  --pred_len 96

5.10 Resume Interrupted Evaluation

python run.py \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --resume_eval

Skips completed (non-NaN) entries, only runs missing/failed combinations.

---

6. Parameter Lists (Space-Separated)

The following arguments accept multiple values (space-separated), automatically expanding to run all combinations:

Example: --refiner_input xy all --update_rule plain bayesian → 2×2=4 variants.

---

7. Output Layout

7.1 Detailed Per-Dataset CSVs

results/details/single_dataset/<dataset>/results_csv_<dataset>_<model>_<Refiner>_<suffix>.csv

Contains baseline and refined rows with MAE/MSE/MAE_raw/MSE_raw.

7.2 Summary CSVs (Aggregated)

results/results_csv_<Refiner>_<suffix>_pred<Len>_all_csv_dataset_mse.csv
results/results_csv_<Refiner>_<suffix>_pred<Len>_all_csv_dataset_mae.csv

Format:

• Rows: datasets (ETTh1, ETTh2, ...) + Models Avg.
• Columns: each base model (Chronos-2, Moirai-2, ...) with Value and Change (%)
• Change = (refined - baseline) / baseline × 100 (negative = improvement)

Suffix examples:

• Bay_xy_bayesian_buf3000_batch256 → proposed method
• Bay_xy_plain_buf3000_batch256 → plain update ablation
• Linear → baseline refiner (no suffix)
• router → baseline refiner with router enabled

7.3 Prediction Length Average

When --pred_len has multiple values:

• ..._pred_avg_all_csv_dataset_mse.csv → average across all pred_lens

7.4 Logs and Plots

• results/details/logs/ → training logs, loss curves (JSON + PNG)
• results/details/logs/online_training_runs/ → online training specific logs
• results/plots/ → comparison plots
• results/attn_maps/ → attention map visualizations (if --attn_maps enabled)

7.5 Inference Cache

data/model_infer_cache/ → cached base model predictions (reuse with --cache)

---

8. Advanced Features

8.1 Attention Map Inspection

python run.py --model moirai-2 --dataset ETTh1 --attn_maps

Extracts and saves attention weights from supported models (moirai-2, chronos-2).

8.2 Channel-Wise Normalization (Electricity)

Automatically enabled for Electricity dataset:

• Metrics: computed per-channel, then averaged
• Refiner input: normalized per-channel (not global)

Other datasets use global normalization.

8.3 Large Data Channel Mode

Automatically activated for {Traffic, Electricity} when pred_len > 100:

• Reduces memory footprint during evaluation
• Processes channels sequentially for single-channel backends

8.4 Bay Loss Variants

Only for Bay refiner:

python run.py --refiner bay --bay_loss mse      # L2 loss (default)
python run.py --refiner bay --bay_loss mae      # L1 loss
python run.py --refiner bay --bay_loss huber    # Huber loss (delta=1.0)
python run.py --refiner bay --bay_loss huber --bay_huber_delta 0.5  # custom delta

8.5 Ablation Flags

# Disable confidence gate (force c_t = 1.0)
python run.py --refiner bay --force_gate_open

# Disable channel mixing (CI-style channel-independent)
python run.py --refiner bay --no-channel_mix

---

9. Plotting Scripts

9.1 Structure Ablation

python plot/plot_abla_structure.py

Plots 5 variants: proposed, w/o router, w/o channel mixing, w/o bayesian prior, w/o decay buffer.

9.2 Input Ablation

python plot/plot_abla_input.py

Horizontal stacked bar chart: 6 datasets × 5 input modes (xy, all, x, y, e_past).

9.3 Hyperparameter Heatmap

python plot/plot_hyper.py 

Heatmap: buffer size (cols) × batch size (rows) × 5 models.

---

10. Quick Reference

Standard Proposed Method Run

python run.py \
  --dataset all \
  --model all \
  --refiner bay \
  --refiner_input xy \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 256 \
  --pred_len 96 \
  --context_length 520 \
  --device cuda

Compare All Input Modes

python run.py \
  --dataset all \
  --model all \
  --refiner bay \
  --refiner_input xy all x y e_past \
  --update_rule bayesian \
  --online_buffer_windows 3000 \
  --batch 256 \
  --pred_len 96

Baseline with Router

python run.py \
  --refiner dsof ta fas solid elf \
  --baseline_router \
  --dataset all \
  --model all \
  --pred_len 96

Resume After Interruption

python run.py --refiner bay --refiner_input xy --update_rule bayesian --resume_eval

---

11. Notes

• timesfm-2 uses the timesfm Python package from active environment
• All refiners are blackbox: base model runs once, no gradient backpropagation
• Default context length: 520
• Default prediction length: 96