Jev Does Not Play Dice

English | 日本語

Code, recorded outputs, and analysis scripts for probability-output experiments with Jev: fair random draws, Noul (Yes/No) questions, and forecast documents.

Write-up: Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll (English) / Jevはサイコロを振らない (日本語, original).

Results

On the fair random tasks, Jev's Choice output often assigned a high probability to its selected option even though the model had no information about the hidden outcome. The issue is not that Jev failed to predict a random event; the observed accuracy stayed close to chance, as expected. The notable result is that the reported probabilities did not reflect that known uncertainty.

Noul was closer to the reference probability in some conditions, but still over-reported probabilities in the low-probability range. In the forecast-document experiment, Choice also did not preserve the uncertainty stated upstream: documents stating probabilities just below and above 50% produced sharply different reported probabilities.

These results are specific to the prompts and conditions in this repository. They do not establish that Jev probabilities are generally unusable.

Requirements

• Python 3.10+ for offline analysis. No extra packages required.
• Node.js 22.16+ for request previews and API runs.
• A Vercel AI Gateway key for API runs only.

Usage

Run all commands from the repository root.

Analyze recorded results

python3 scripts/analyze.py

Writes CSV, JSON, and Markdown summaries to output/summary/. No API calls; recorded data is not modified. See the saved summaries and figures.

Preview requests

node src/run.mjs --input examples/dice.json
node src/run.mjs --input examples/noul.json
node src/run.mjs --input examples/forecast.json

These examples contain 1, 3, and 2 requests respectively. examples/dice.json is trial d0020 from the 400-trial die sweep, the request shown in the article; its recorded result is 0.83, the median. examples/dice_ordered.json is a fixed-wording die prompt with face_N option keys, which keep the order you write them in (integer-like keys such as "1" are always sorted by JavaScript); its recorded results are lower, about 0.72. Preview is the default and needs neither an API key nor installed npm dependencies. Edit the example JSON to try other inputs.

Run an experiment

npm ci
cp .env.example .env.local
# Set AI_GATEWAY_API_KEY in .env.local.
npm run experiment -- --input examples/dice.json --execute --out runs/my-dice

Only --execute enables billable API calls. Defaults: sequential execution, 20 seconds per attempt, up to 3 attempts per case. Results go to runs/, which is excluded from Git. Use --resume to continue a run; see runner options.

Tests

npm test
python3 -m unittest discover -s tests -p 'test_*.py'
node scripts/build_requests.mjs --check

Tests run offline with mock API responses. For optional figure regeneration, see plotting.

Repository layout

examples/       Small, editable request sets
src/            Runner, validation, and request reconstruction
scripts/        Analysis, plotting, and input generation
data/recorded/  Historical model outputs
data/requests/  Reconstructed requests
data/summary/   Precomputed summaries
figures/        Japanese and English charts
tests/          Offline tests
docs/           Methods and usage notes

Documentation

Methods and limitations · Data format and provenance · Running experiments · Verification

Noul uses type: "boolean" in the Vercel AI SDK. Recorded results and new runs are kept separate. Forecast-document percentages are stated values, not independently verified event probabilities.

The historical responses do not themselves record an exact model revision. However, the experiments were run during the launch window when jev-1.13.0 was the only publicly available Jev version. Independent project; not affiliated with TypeSafe or Vercel.

License

MIT