Decision Model Testing

Benchmark the classification performance of fine-tuned Laya, untrained Laya, and the OpenAI Decisions API across several application datasets. The repository consolidates evaluation scripts, the GPU training implementation, dataset inputs, original result artifacts, and consistently formatted metrics. OpenAI runs use gpt-6-luna; the completed GPU runs use laya-multilingual as base model.

Scripts and data paths are relative to this repository. Credentials and GPU endpoint settings are supplied through environment variables.

Datasets

• Banking77: banking customer intent classification, with independently sampled 25- and 50-category subsets and the full 77-category set. Original training and test partitions are preserved.
• Contract NLI: contract/hypothesis classification into contradiction, entailment, or neutral. We tested the Contracts B version of the dataset. Laya uses an 8,192-token sequence budget, full hypotheses, and audited contract truncation where necessary.
• DynaSent: DynaSent v1.1 Round 1 Yelp sentiment classification into positive, negative, or neutral. Laya trains on 80,488 majority-labeled examples; mixed and no-majority gold labels are excluded. Human vote counts are preserved, but this run uses hard labels rather than probabilistic training targets.
• SNLI: premise/hypothesis classification into entailment, neutral, or contradiction. Rows with label -1 are excluded before uniformly sampling 100,000 train and 2,500 test examples with seed 42.

High-level results

Scores are macro F1 (0 to 1). The best-scoring model in each row is bold.

All model runs with retained metrics.json and predictions.jsonl files appear in benchmark_summary.csv and benchmark_summary.json.

Result files

Every model result folder contains the same three files:

• predictions.jsonl: per-sample gold/predicted labels, status, correctness, probabilities, and original sample ID.
• metrics.json: accuracy, macro/weighted F1, per-class precision/recall/F1/support, class order, confusion matrix, failure counts, and the repository-relative result filename.
• confusion_matrix.json: matrix with explicit axis/class ordering and separate failed/refused counts by true class.

No missing predictions or scores were fabricated. OpenAI folders additionally contain an Excel report, results.xlsx. Banking77 training configurations are kept in experiments/PolyAI_Banking77/configs/, outside the model result folders.

Banking77 25/50/77, Contract NLI B, DynaSent, and SNLI test runs also contain sample_comparison.csv, joining OpenAI, fine-tuned Laya, and baseline Laya labels for the same sample. Sample IDs and gold labels were checked for agreement. Banking77 OpenAI's original 1-based IDs are normalized to the GPU's 0-based IDs; Contract NLI API IDs are normalized to test_<index>. Input text is in each experiment's dataset files; predictions avoid duplicating large contract texts. DynaSent predictions retain human vote counts; SNLI predictions retain the premise and hypothesis. Both datasets additionally report NLL, multiclass Brier score, and 15-bin ECE against hard gold labels.

SNLI's six disagreement examples and five-repeat checks on three selected examples are retained in its runs/ folder. Repeatability folders contain predictions.jsonl, predictions.csv, and summary.json, not full-test metrics, and are excluded from the high-level benchmark summary. Ignored .work/ folders retain local GPU restart state outside published results.

Run uv run python tools/summarize_results.py to verify recorded metrics and rebuild benchmark summaries and sample comparison files.

Running experiments

From this repository's root:

uv sync

OpenAI credentials are read from MY_OPENAI_API_KEY, then OPENAI_API_KEY. Alternatively, evaluators support --key-file containing literal environment assignments. Never commit credentials.

# Banking77
uv run python experiments/PolyAI_Banking77/evaluate_openai_decisions.py \
  experiments/PolyAI_Banking77/data/category_subsets/25_categories/val.csv \
  --output-dir experiments/PolyAI_Banking77/runs/25_categories/new_openai_results

# Contract NLI B test
uv run python experiments/CONTRACT_NLI/scripts/evaluate_openai_decisions.py \
  experiments/CONTRACT_NLI/data/contract_nli_b/test.parquet \
  --output-dir experiments/CONTRACT_NLI/runs/new_openai

# DynaSent test
uv run python experiments/DYNASENT/scripts/evaluate_openai_decisions.py \
  experiments/DYNASENT/prepared/test.csv \
  --output-dir experiments/DYNASENT/runs/new_openai

# SNLI 2,500-sample test subset
uv run python experiments/SNLI/scripts/evaluate_openai_decisions.py \
  experiments/SNLI/prepared/test.csv \
  --output-dir experiments/SNLI/runs/new_openai

All API evaluators use eight workers and overwrite their chosen output. Use new output paths to preserve historical runs.

For GPU training/evaluation, set LAYA_TRAINER_URL and LAYA_TRAINER_API_KEY securely in your environment, then use the dataset's run_laya.py:

uv run python experiments/PolyAI_Banking77/run_laya.py \
  --config experiments/PolyAI_Banking77/configs/25_categories.json \
  --train experiments/PolyAI_Banking77/data/category_subsets/25_categories/train.csv \
  --val experiments/PolyAI_Banking77/data/category_subsets/25_categories/val.csv \
  --output-dir experiments/PolyAI_Banking77/runs/25_categories/new_laya_results

--job-type baseline_only omits training. --job-type evaluate_only evaluates the checkpoint in the config. --resume-job <job_id> resumes monitoring an existing job without submitting another. Model weights stay on the GPU; downloaded artifacts contain metrics and predictions. For DynaSent and SNLI, scripts/run_laya.py handles train+dev followed by a separate frozen-checkpoint test evaluation and baseline comparison:

uv run python experiments/DYNASENT/scripts/run_laya.py
uv run python experiments/SNLI/scripts/run_laya.py

Those scripts resume the existing jobs from ignored .work/ metadata by default. Use a new work directory and result directory for a new experiment. See each dataset's README for preparation commands, split policies, and details. The copied GPU implementation is in laya_trainer/core/train.py, laya_trainer/core/evaluate.py, and laya_trainer/service/worker.py. This is a snapshot of the available local trainer source, not a recorded version pin proving the exact code/dependency versions used by every historical job. It requires a separate GPU environment with torch, transformers, and Laya; these are not installed by the local API-only uv sync.

Interpretation and limitations

• Fine-tuned Laya has task-specific supervised training; OpenAI is evaluated without task-specific fine-tuning. These are deployment comparisons, not matched-training comparisons.
• All paired model results use matching test samples within each experiment.
• The GPU trainer computes per-epoch metrics and fits temperature on the supplied evaluation set. Contract NLI's train+test run is therefore not a strictly held-out test protocol. A cleaner protocol uses separate training/calibration data and evaluates the frozen checkpoint on test only.
• DynaSent and SNLI use that cleaner protocol: official dev data for per-epoch evaluation and temperature calibration, then a frozen checkpoint on test. DynaSent uses 3,600 dev rows; SNLI uses 9,842 labeled official dev rows. Neither dataset uses test labels for training or calibration.
• DynaSent hard-label training does not consume its human vote distributions. No selected DynaSent or SNLI input was truncated.
• Banking77 subsets are independent, not nested. Changes in scores across category counts reflect both category count and category composition.
• OpenAI refusals are preserved and count as incorrect in overall accuracy and as false negatives for gold-class recall. Confusion matrices record those separately instead of silently dropping them.
• Contract NLI preparation truncated 34 training contracts; no test sample was truncated. Truncation can remove decisive evidence.
• Secrets and model weights are excluded. Dataset redistribution rights and intended repository visibility should be reviewed before publication.

Next steps

Add more models

• Jev
• Strands Decision (Amazon)
• Clef-Flash (Cloudflare)

Add non-English datasets

Extend the evaluations to non-English datasets to compare multilingual performance.

Dataset sources

• Banking77: Original repository | Hugging Face | Paper | License: CC BY 4.0
• ContractNLI: Dataset website | Original repository | Paper | License: CC BY 4.0
• DynaSent: Original repository | Hugging Face | Paper | License: CC BY 4.0
• SNLI: Dataset website | Hugging Face | Paper | License: CC BY-SA 4.0

Using this repository's results and findings

Please cite Decision Model Benchmarks when using its results or findings, and cite the underlying datasets separately. For reproducibility, optionally record the experiment and commit used.

Suggested BibTeX entry (replace the access date placeholder):

@misc{decision_model_benchmarks,
  author = {Santhosh, Roshan},
  title  = {Decision Model Benchmarks},
  year   = {2026},
  url    = {https://github.com/rsk2327/decision-model-benchmarks},
  note   = {Accessed: YYYY-MM-DD}
}

This acknowledgment guidance does not grant additional rights to upstream datasets or establish a license for repository code or results.

Structure

DECISION_MODEL_TESTING/
  README.md
  pyproject.toml
  benchmark_summary.csv
  benchmark_summary.json
  source_artifact_hashes.json
  experiments/
    PolyAI_Banking77/
      evaluate_openai_decisions.py
      create_category_subsets.py
      run_laya.py
      configs/{25,50,77}_categories.json
      data/
      runs/{25,50,77}_categories/
        openai/
        laya_baseline/
        laya_finetuned/
        sample_comparison.csv
    CONTRACT_NLI/
      scripts/
        evaluate_openai_decisions.py
        prepare_laya_data.py
        run_laya.py
      data/contract_nli_b/
      prepared/
      runs/{openai,laya_baseline,laya_finetuned}/
    DYNASENT/
      README.md
      scripts/{prepare_laya_data,evaluate_openai_decisions,run_laya}.py
      data/
      prepared/{train,dev,test}.csv
      prepared/manifest.json
      configs/train.json
      runs/{openai,laya_baseline,laya_finetuned}/
      runs/sample_comparison.csv
    SNLI/
      README.md
      scripts/{prepare_laya_data,evaluate_openai_decisions,run_laya}.py
      data/
      prepared/{train,dev,test}.csv
      prepared/manifest.json
      configs/train.json
      runs/{openai,laya_baseline,laya_finetuned}/
      runs/sample_comparison.csv
      runs/{openai_repeatability,laya_baseline_repeatability}/
  laya_trainer/
    core/
    service/
    runners/
    GPU_server_start.sh
    CONFIG_SCHEMA.md
  tools/
    run_laya_job.py
    classification_reports.py
    summarize_results.py