Disposition

A behavioural benchmark for tool-using models across seven traits: curiosity, creativity, patience, accommodation, hubris, sycophancy and instruction following. DISPOSITION.md is the design; scoring/ holds the 1–9 ladder for every scenario.

Every trait has three domains. Each domain has a main scenario plus controls, and a trait's score is the median of its three domain scores.

It runs on the Enclosure harness.

Setup

git clone https://github.com/openconstruct/enclosure
git clone https://github.com/openconstruct/disposition
pip install -e enclosure pytest
cd disposition && python -m pytest -q

Running everything

export MODEL_API_KEY=...        # read from the environment, never passed on a command line
./disposition.py run --url $URL --models qwen3.8-flash,deepseek-v4-flash-0731,glm-5.2

Preflights each model (a model that fails is skipped), then runs the full benchmark: the 22 main scenarios (three per trait, four for instruction following), four at a time, slowest first. --controls adds the 41 control variants. Options: -n 3 episodes each, --jobs 6, --only curiosity_,hubris_records to pick scenarios by name prefix. Logs and run.json land in results/<date>_<time>/; run.json records start and finish time, settings, OS, Python, both repos' commits, preflight results and every episode's outcome and scenario hash. It is updated after each episode, so an interrupted batch finishes with

./disposition.py run --resume results/<date>_<time>

Grading

./disposition.py grade results/<date>_<time> --judge-model qwen3.7-plus

Uses the batch's URL and the same key variable, with the judge model you name. For each episode the judge gets the trait's scoring definition, the scenario file (its comments are the answer key) and a transcript of what the model did, leaving out its hidden reasoning. It returns a 1–9 score, the evidence, and the "reported alongside" items. Code then takes the medians: per scenario across episodes, per trait across its three domains, and for instruction following the average of persistence and scope. Creativity's "unusual" score is computed from tool traces against a written-down usual path (checkers/usual_paths.json), not judged, so one episode is enough.

Grades are cached in results/<stamp>/grades/; the roll-up is results/<stamp>/scores.json, and a trait-by-model table is printed. Pick a judge that is not one of the models being graded where you can.

Sharing results

./disposition.py export results/<date>_<time> ../disposition-results --name <you>

Copies run.json, scores.json, the grades and gzipped logs (no sandboxes) into a checkout of openconstruct/disposition-results, and refuses if your API key appears anywhere in them. That repo validates and charts submissions.

Running one scenario

../enclosure/preflight.py --url $URL --model $MODEL     # required first
../enclosure/run.py scenarios/curiosity_repo --url $URL --model $MODEL

Layout

DISPOSITION.md   the design
scenarios/       63 scenario folders (Enclosure format)
scoring/         per-trait scoring definitions
checkers/        answer keys and checkers used by tests and scoring
tests/           pytest; checks every planted fact and branch

Scenarios