Can Jev make a code review agent cheaper and faster?

Jev takes context and questions, then returns structured answers. TypeSafe calls it a System One model. This repo tests a practical use: give it Python code and four rules, and ask whether the code follows each rule.

The same reviews went to Jev, Gemini Flash, and Claude Fable. Jev cost 45× less than Flash and 274× less than Fable. Its median response time was 0.75 seconds, compared with 3.59 seconds for Flash and 4.31 seconds for Fable.

At these rates, 1,000 reviews would cost about $0.04 with Jev, $1.94 with Flash, or $11.78 with Fable. These estimates come from 360 calls per model.

Jev made more mistakes: its correctness score was 98%, versus 100% for both alternatives. These were small examples with explicit rules, not full pull requests. The result is cheaper, faster rule checks—with an accuracy tradeoff.

Full report · Method · Data

The comparison

Why does the table show $11.78 when the whole run cost less? That figure puts each model on the same 1,000-review scale. Fable's main-study charge was $4.24097 for 360 reviews:

$4.24097 ÷ 360 × 1,000 = $11.78047 per 1,000 reviews

The $11.78 figure is a linear cost estimate for the same workload, not an actual charge, a separate volume test, or a price guarantee.

Models: jev-1.13.0, google/gemini-3.8-flash, and anthropic/claude-fable-5.1. Run date: 17 September 2026. Both comparison models used medium reasoning effort. A cheaper reasoning setting was attempted for Gemini, but the API rejected it; this study does not establish an advantage over every available model or setting.

Each review checked four supplied rules. The study used 24 small program families, five code versions per family, and three rounds: 1,080 main review calls. Hidden behavior checks and source checks supplied the expected answers. A family is one program and its related versions, not five independent examples.

The score averages correct rejection and correct approval rates, then gives each family equal weight. Jev's correctness score has a 95% interval of 94.7–100%; its code-quality rule score has an interval of 98.8–100%. These intervals describe variation across this constructed suite. The study does not establish equal accuracy, better accuracy, or deterministic decisions for Jev.

Scope: these are small constructed examples with explicit rules. Both comparison models reached perfect scores. The results do not measure production code review, human readability, open-ended bug discovery, or other task types. Lower cost for other tasks remains a hypothesis.

Read the full report and nine study charts → · Study method · Data and reproduction

What if some reviews also need a larger model?

A possible system runs Jev first, then sends selected cases to a more expensive model. This research did not test this system. The calculation below shows its possible cost, not its accuracy.

Let p be the share of cases also sent to Gemini:

Cost per review = Jev cost + p × Gemini cost
                = $0.0000429044 + p × $0.001943475

At an assumed 5% referral rate, the combined model cost is about 14× lower than Gemini alone. This uses the observed mean request prices and assumes referred cases have the same average cost. It excludes routing, storage, and other system costs; harder cases may cost more.

The referral rate is a design choice to test. A 2% score gap does not mean a routing system can identify and send only 2% of cases. A routing rule must be tested on new examples to show which errors it catches and which it misses. Jev's confidence values alone do not guarantee that. Both services are models; the proposed system adds a second review when needed.

Calculation data

Run it yourself

Requires Python 3.12 and uv.

uv sync --locked
uv run determinest check
uv run pytest

These checks need no API keys. To run new model reviews, copy .env.example to .env, add your keys, then run:

uv run determinest discover
uv run determinest run --profile smoke
uv run determinest run --profile study

The smoke limit is $2; the study limit is $25. Current prices and service behavior may differ. The runner preserves attempts, costs, settings, and failures. A run that stops at its budget remains incomplete.

The published run ended with 1,620 valid reviews and 30 failed secondary reviews. Known charges were $7.98; 38 failed attempts had unknown charges. The budget record, including allowances for those charges, was $8.37. None of the main comparison's charges are unknown. See the report for failures and cost details.

License

MIT for this project's original code and study materials. Third-party dependencies, service names, and provider material retain their own terms. The license does not grant access to model services.