Jev vs. LLMs on Reddit's "Am I the Asshole?"

Bottom line: In the original seven-setup benchmark, Jev came second, behind Sonnet 5 (Brier score 0.369 vs. 0.344; lower is better). Jev's median call was 6.3× faster than Sonnet's, and 62× cheaper. That's fast, but not the "40x-200x faster" TypeSafe claims.

The chart includes two later Jev follow-ups. I did not retrain Jev itself: a small logistic regression learned from its answers to 300 labeled 2023 posts, then adjusted its probabilities on the 770 test posts. That gives Jev the lowest score shown (0.337), but Sonnet's 0.344 is unadjusted, so it is not a like-for-like win. A separate analysis gave every model the same kind of adjustment; Sonnet still led by 0.007, an inconclusive gap.

Jev is TypeSafe's "System One" model. It doesn't write text: you give it a situation and a question, and in one fast call it returns a probability for each answer. TypeSafe says it matches LLMs on quick judgments like this. I tested jev-1.13-20260917 on posts from Reddit's r/AmItheAsshole, where readers vote on who was at fault in a conflict:

The main benchmark compared Jev, Sonnet 5, GPT-5 nano and two local open-weight models on 770 posts from 2025. Every model gave a probability for each of the four verdicts and was scored against Reddit's verdict.

I have no affiliation with TypeSafe and paid for every call myself (about $2.75 in total).

What counts as "correct"

Whether someone is the asshole is an opinion, so there is no true answer here. This benchmark measures something narrower: can a model predict the verdict Reddit gave? The target is the post's official verdict flair, which AITA sets from the top-voted comment. A model is "right" when its top verdict matches that flair. The main score, the Brier score, also gives partial credit: putting 70% on Reddit's verdict scores better than putting 30% on it.

Reddit's verdict is itself noisy, because one top comment decides it even when other commenters disagree. On 704 of the 770 posts, the flair matches what most commenters said (weighted by upvotes); on the other 66, it doesn't. Scoring only the 704 clearer posts tells the same story: Jev 63.4% vs. Sonnet 64.3% plain accuracy (details). "Not enough info" posts are left out, since that's not a verdict on anyone.

Part 1: choosing Jev's question before the benchmark

In one call, Jev can answer one multiple-choice question (pick one of the four verdicts) or several yes/no and rating questions. Before the benchmark, I tried eight ways of asking. None was clearly better than simply asking directly: "which verdict would the subreddit reach?" So the main benchmark uses that direct question.

On 200 older posts, asking directly got 53.0% right. Asking eight yes/no questions about the facts, then turning the answers into a verdict, got 41.5% with a hand-written rule and 51.5% with a rule learned from data. Ten questions, including the direct one, got 55.0%: four more posts right than direct, which could easily be luck (p = 0.22).

Part 2: Jev vs. LLMs on 770 posts

How to read the table:

• Brier score grades all four probabilities, from 0 (perfect) to 2 (worst). Lower is better. Being confidently wrong costs more than hedging.
• Weighted: on Reddit, 74% of verdicts are NTA and about 4% each are ESH and NAH. The sample has about twice as many ESH and NAH posts as that, so there are enough of them to measure. Weighting puts each verdict back at its real share: an NTA post counts about 1.6× as much as in a plain average, an ESH post about half as much.
• Top-1: how often the model's top verdict matched Reddit's, weighted the same way.
• Macro recall: how often each of the four verdicts was caught, averaged. A model that always says NTA gets 25%. This rewards catching the rare verdicts.
• Brackets are 95% confidence intervals (CI), from resampling posts. They show how much a score depends on which posts were picked, not how much a model varies between runs: each model ran once, and even at temperature 0 a restarted GPT-5 nano run repeated only 64 of 103 answers.

Jev's two-question setup combines two yes/no probabilities into four verdict probabilities. It is separate from the later five-question check.

The two italic Jev rows use separate rules trained on 300 older posts. The seven original model rows use answers without adjustment. All runs used the same 770-post sample. The adjustment and five-question run were chosen after the original results were known, so this is not a fresh holdout or a like-for-like win over raw Sonnet.

• Among unadjusted setups, Sonnet 5 came first and Jev direct second. Sonnet's lead is 0.025 (95% CI 0.003 to 0.046). That is outside noise on its own, but not after correcting for five comparisons with Jev (Bonferroni), so call it borderline. On another score, log loss, Sonnet's lead is clearer and survives that correction. Without weighting, there's no clear difference (Jev 0.556, Sonnet 0.563). Jev beat both GPT-5 nano settings and both local models.
• Jev was 6.3× faster than Sonnet, not 40–200×. Its median call took 0.39 s, inside TypeSafe's 70–500 ms claim, and it was 62× cheaper than Sonnet. Against GPT-5 nano it was 3.9× faster and 1.5× cheaper than minimal effort, and 12× faster and 4.5× cheaper than low effort. TypeSafe compared against a model reasoning at its default level; the chat models here did little or no reasoning, which makes them much faster.
• Guessing from base rates is hard to beat. Always saying NTA is right 74% of the time. The "no model" row gives every post Reddit's base rates. Only Sonnet and Jev clearly beat it (by 0.070 and 0.046). Qwen was slightly better on paper (0.410 vs. 0.415), but that could be noise (95% CI: from 0.033 better to 0.022 worse than base rates). Every model missed more than half of the ESH and NAH posts.

Local models ran as 4-bit MLX builds on one laptop; Qwen's 4 malformed answers count as wrong. More metrics, confusion matrices and calibration are in the full results.

What changes with trained adjustments?

Jev itself was not retrained. A small regression learned to turn each setup's answers into four verdict probabilities. On 300 older 2023 posts, each scored by a rule trained on the others, five questions scored 0.312 against 0.320 for adjusted direct. The difference was inconclusive. I then trained separate rules on all 300 posts and applied them to the 770-post test for the two italic Jev rows above. On those 770 posts, five questions scored 0.009 worse than adjusted direct (95% CI: 0.001 better to 0.019 worse). Neither adjusted setup correctly picked ESH or NAH. The five-question run was chosen after the original test results were known, so these are exploratory results on the same posts, not a new holdout. Development check, question wording, and 770-post follow-up.

For a like-for-like adjustment check, I also fitted the same kind of regression for every original configuration. Each post's adjusted answer came from a rule trained on the other nine of ten folds of the 770-post set; no new model calls were made. Sonnet scored 0.332 and Jev direct 0.339, a lead of 0.007 with an interval that includes zero. The adjustment changed verdict choices and no model picked ESH or NAH afterward. This analysis also used already-known test results, so it does not replace the original ranking. Plan and full results.

What this does not show

• Who is actually right. Only how well each model predicts Reddit's verdict (see above).
• Label-only use. Every model had to give four probabilities, Jev natively and the chat models as JSON. Asking a chat model for one label would change its accuracy, speed and cost.
• Full-size local models. Qwen and Gemma ran as 4-bit builds with reasoning off. The prompts also differ slightly: Jev is asked which verdict the subreddit would reach, the chat models to predict the official flair.

Method details

• Posts: 770 from UC Berkeley D-Lab's 2025 dataset (CC BY 4.0, pinned revision): 365 NTA, 285 YTA, 65 ESH and 55 NAH. Models saw only the title and text. Posts that state their own verdict ("EDIT: seems I'm NTA") were removed. Neither the 200-post selection set nor the 300-post follow-up overlaps the benchmark.
• Plan: the protocol lists four deviations from it, including a parser fix that restarted two runs. Neither the plan nor the exact code that ran was committed before the runs, so their timing can't be proven from git; the protocol lists what can still be checked.
• Prompts: in client.py and questions.py, pinned by a test and checked against billed token counts by a script.
• Intervals: 95% bootstrap over posts within each verdict. They exclude run-to-run variation: at temperature 0, a restarted GPT-5 nano run repeated only 64 of 103 answers.
• Latency: one laptop in UTC+3, via OpenRouter, one evening. Jev is served from the US West Coast, so its times include a long network trip.

More: methodology, run logs.

Try it

Python 3.10+, standard library only. Clone the repo and run from its folder.

Anything that asks Jev live needs an OpenRouter API key, because this repo calls Jev through OpenRouter. Create one at openrouter.ai/keys and add a little credit; a dollar covers thousands of calls. OpenRouter's terms currently set a $5 minimum credit purchase, in US dollars, if you need to top up. That is prepaid credit, not the cost of one run. Then copy .env.example to .env and paste the key in. Rebuilding the results from the logs needs no key.

Judge a real AITA post

This pulls a random real post, with Reddit's verdict, from the same Hugging Face dataset, and asks Jev live (two calls, about $0.0001). It shows both setups for comparison: Step 1 is the direct question used in the benchmark; Step 2 is the two-yes/no setup, which scored worse across the benchmark.

python -m jevbench.show --random

One live run, for illustration (not saved; numbers vary between calls):

━━ POST 1ilu5ya ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3071 chars
  AITA for telling my boyfriend not to bother coming over to my house anymore?
  …
  Forum verdict: NTA (Not the asshole -- the other party is in the wrong)

━━ STEP 1 · Direct question ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 553 ms · $0.000047
  Answered: choice NTA, confidence 0.71

  NTA   0.78  ███████████████████████████████  ← picked · forum verdict
  YTA   0.14  ██████
  NAH   0.04  ██
  ESH   0.04  ██

  Jev picked NTA · forum said NTA · ✓ right · Brier 0.071

━━ STEP 2 · Two yes/no questions ━━━━━━━━━━━━━━━━━━━━━━━━━━━ …

The post may be one of the 770 benchmark posts; its answer is fetched fresh either way. To judge your own post instead, pass the text (first line is the title), or pipe it in with --text -:

python -m jevbench.show --text "AITA for not lending my car to my sister?
She crashed my last one and never paid for the repairs."

Rebuild the results

No key or network needed. This regenerates the main benchmark tables and headline chart from the committed logs:

python -m jevbench.report runs/final-*.jsonl \
  --priors data/final-ucb-2025.source.json \
  --chart docs/headline.svg --five-followup runs/followup-yesno5-2025.jsonl

It rewrites docs/results.md without one section (Reddit's verdict against the comment vote), which needs the posts from the next step. The 300-post development check rebuilds from its logs:

python -m jevbench.steps fit runs/diagnostic/steps-dev-*.jsonl

The adjustment check for every model also rebuilds without a key:

python -m jevbench.recalibrate

Replay a benchmark post

To see a post from the benchmark next to Jev's logged answer and Reddit's verdict, first rebuild the 770 posts. This downloads the dataset from the same Hugging Face API --random uses, refuses it if it doesn't match the pinned hash, and draws the same sample:

python -m jevbench.sample_2025 --fetch --raw data/ucb-2025-raw.jsonl --out data/final-ucb-2025.jsonl
echo "2f8cae7c7bebe3c259efbe69664b463919a9db5b0db386d23a79d8892f82479d  data/final-ucb-2025.jsonl" | shasum -a 256 -c

Then:

python -m jevbench.show 1hvkncw

Leave out the id for the first post, add --live to ask Jev again, or --short to print only the post's first lines (this works with --random and --text too). With the posts rebuilt, adding --source-raw data/ucb-2025-raw.jsonl to the report command restores its missing section.

Run a new benchmark

python -m jevbench.bench --help. The runner has a spending cap, and each run's manifest records the exact prompts.

jevbench/   runner, clients, metrics, report, viewer, step setups, recalibration (stdlib only)
scripts/    prompt verification against billed tokens (needs tiktoken)
runs/       final logs; diagnostic/ holds later checks and pilots; dev-2023/ the first round
docs/       results, methodology, protocol and charts
tests/      pytest suite, no network