this-that-complex-decisions

1,710 decisions where the answer follows from a stated policy applied to a state, and where no single field of that state gives it away.

1,710 questions   19 decision types   40 domains   chance rate 0.258

Each row is a state, a question, a closed set of options, and the index of the one option the policy selects. The answer is determinate: given the state and the policy there is exactly one correct choice, and it does not depend on anyone's judgement.

What is in a row

{"id": "authority_conflict-0000",
 "family": "authority_conflict",
 "state": "candidates:\n  staff-02:\n    skill_match_pct: 87\n    hours_this_week: 56\n ...",
 "question": "Who takes the shift?",
 "options": ["staff-02", "staff-04", "staff-01", "leave the shift unfilled"],
 "answer_index": 1,
 "n_options": 4}

Score as accuracy against answer_index. A system that returns nothing is wrong, not excused — that is the rule every number below was produced under. Option counts vary by row, so the chance rate is the mean of 1 / n_options rather than a single fraction.

Results

All nine systems on all 1,710 questions, 90 per decision type.

By decision type

One decision type defeats almost everything

knapsack_subset asks for the group of items that fits a budget and maximises value. Greedy selection fails on it about half the time, so it cannot be answered by taking the best-looking option. Eight of the nine systems land between 0.067 and 0.267 against a chance rate of 0.084. The ninth is 1.88 B parameters and returns no tokens.

Where a small model wins and where it does not

Each bar is this-that-model-1.1 minus the better of claude-opus-5 and gpt-5.6 on that decision type. Four types go the small model's way, two are a tie at 1.000, and three more are inside a tenth. The nine that go the other way are worth as much attention: judging that a state does not contain enough to decide (−0.389) and iterating a cascade to its fixed point (−0.300) are where the gap is widest.

What one decision costs

The horizontal axis is not like for like and the chart says so: a hosted system's figure is its published price, which must cover serving and margin; ours is electricity at 80 W and $0.30/kWh for the 30.9 ms the card is busy. Read the distance as one order of magnitude rather than five. What it does show is that one of these is a rounding error on a machine you already own.

What the table says that the ranking does not

One decision type defeats almost everything. On knapsack_subset — choose the group that fits a budget — eight of the nine systems land between 0.07 and 0.27 against a chance rate of 0.084. The ninth, a 1.88 B model that generates no tokens, reaches 0.678. Whatever that row measures, it is not what model scale usually buys.

Scale is not the axis. deepseek-v4-pro scores below deepseek-v4.1-flash. glm-5.3 at 0.652 sits above kimi-k3 at 0.522. And a 1.88 B model sits above every hosted system except the two largest, at a thirtieth of their latency.

Every system has a floor, and they are not the same floor. weighted_aggregate holds the best system to 0.689 and insufficient_information holds this-that-model-1.1 to 0.444 while claude-opus-5 reaches 0.833 there. Both ask for something the ranking hides: weigh several numbers against each other, or judge that the state does not contain enough to decide at all.

value_of_information separates cleanly. Chance is 0.050 — the options are field names, so guessing is close to hopeless — and the systems split into six at or near 1.000 and laya at 0.033.

1.0's row is zero-shot. This benchmark did not exist when that checkpoint was trained. The distance from 0.406 to 0.775 is the same architecture with different training data.

Licence

MIT.