certo — synthetic decision dataset

Synthetic decisions with a known, exact answer distribution, for training and evaluating calibrated decision models. Each example is generated from a conditional naive-Bayes evidence world, so the posterior over the answer is computed in closed form — you can grade a model against the true probabilities (posterior fidelity), not just accuracy.

Part of certo · project page: https://altslate-labs.github.io/certo/

Schema (JSONL, one example per line)

{
  "world_id": 0,
  "params": { "K": 8, "F": 12, "V": 6, "rho": 0.5, "prior_alpha": 0.7 },
  "prior":      [0.20, 0.05, ...],                       // per-example base rates (in the state)
  "state":      [ {"f": 3, "v": 7}, ... ],               // revealed evidence (feature, value)
  "state_text": "prior: c0=0.20 ... | f3=v7 | ...",      // serialized input
  "y_true": 5,                                           // sampled true class (eval only)
  "posterior": [0.02, ..., 0.66, ...],                   // EXACT P(class | evidence) — the answer key
  "questions": [
    { "type": "choice",  "options": [...], "target": [...] },   // named classes + OUT (= complement)
    { "type": "binary",  "subset": [...],  "target": 0.83 },     // P(y in subset)
    { "type": "score",   "class_to_level": [...], "target": [...], "scalar": 3.9 }
  ]
}

• posterior is the exact Bayesian answer given the shown prior + state — the target to grade against.
• Each question is a marginalization of that posterior (choice with an OUT/none option, binary predicate, ordinal score).
• train.jsonl (40k, seed 0) · val.jsonl (5k, seed 1).

Intended use

Train with soft (distributional) targets to get calibrated probabilities; grade with KL/TV to posterior. See the certo repo for the generator (datagen.py, reproducible + self-verified), models, and the training recipe.

Note: this is a synthetic dataset for studying and building calibrated decision models — not real-world data. License: MIT.