Agent Review Studio

Agent Review Studio is a local-first quality and evaluation workbench for AI agents and agent harnesses. It turns real workflows into repeatable tests, helps people separate strong agent behaviour from failures, records human corrections beside the evidence, and shows whether a new harness version is genuinely better than its baseline.

Use it before or alongside agent training and fine-tuning: define evaluation cases, review complete execution traces, label claim and action failures, compare versions, and export trusted improvement data. Agent Review Studio prepares evidence and datasets that can guide prompt, retrieval, tool, workflow or later model-training work; it does not itself change model weights.

Fresh installations contain no preloaded agent, workspace or evaluation runs. Each operator creates and names the agent, harness, workflow or service they want to evaluate. Chaser Agent exists only in the owner's personal browser data and as an optional importable case-study pack; it is not a product default.

Live application: agent-review-studio.chaseintech.chatgpt.site

Version 1.5.0 is the current public repository release. The hosted application is a separate deployment surface, so its displayed version should be checked before treating it as equivalent to the repository release.

Version 1.5.0 makes that evaluation loop agent-agnostic from first launch. A new operator starts with an empty workspace screen and names the system they are evaluating; no Chaser Agent identity or data is injected. Existing personal Chaser installations migrate into ordinary deletable local workspaces. Workspace setup captures the agent or harness purpose and evaluation goal; the sidebar accordion keeps Pages and Runs inside each workspace; and deterministic checks, probabilistic evaluators and human decisions remain visibly separate.

What process is this?

This is agent evaluation, human labelling, evidence curation and harness refinement. It prepares trusted improvement data; it is not automatic model-weight fine-tuning.

The review output can improve:

• prompts and system instructions;
• workflow and orchestration logic;
• retrieval, evidence and provenance handling;
• tool-selection and approval policies;
• memory admission rules;
• golden cases, regression suites and future training datasets.

Model fine-tuning may later consume carefully selected review data, but this application currently changes neither model weights nor imported source artifacts.

Can this train an agent?

It can fully run the evaluation and improvement-data workflow for an agent or agent harness:

1. define a repeatable test case and preserve its source;
2. run or import a versioned harness result;
3. inspect claims, evidence, actions and the execution trace;
4. combine deterministic evaluator results with human labels and scores;
5. preserve corrections as immutable reviewed examples;
6. compare a candidate with a baseline and apply regression gates;
7. export the approved evidence for prompt, tool, retrieval, workflow or training-data work.

Agent Review Studio does not itself update model weights. Weight fine-tuning still needs a separate training system, an approved dataset with target outputs, model configuration, compute and a fresh evaluation pass. This distinction prevents a completed review from being mistaken for a completed training job.

First review: seven steps

1. Open Overview and choose the workspace, evaluation session and next run.
2. Use Understand the task to read the goal, retained source, provenance and run boundary.
3. Use Verify the agent's work to inspect each claim candidate beside its exact evidence and surrounding context.
4. Assign categorical labels to every claim. Add a correction whenever a failure label requires one.
5. Confirm actions, memory, uncertainty and the execution trace, then score the five run-level quality dimensions once.
6. Finish the review to create an immutable revision; re-review creates another linked revision rather than overwriting it.
7. Use Compare and Insights to decide whether the candidate improves on the baseline and whether it passes the configured gate.

The complete beginner walkthrough is in docs/OPERATOR_GUIDE.md. The in-product Guided tour visits these same controls and can be restarted from Settings.

Working capabilities

• A prominent workspace switcher that lists every active workspace with a directly labelled Delete action, archived-workspace access and guided creation.
• Reversible archive and restore plus guarded permanent deletion that requires the exact workspace name.
• An accordion-style multi-workspace stack where every workspace owns its Pages and Runs tree, remembers its view and collapses when another opens.
• Visible three-dot and right-click workspace actions for rename, detail editing, archive and exact-name-confirmed deletion.
• A minimisable, drag-resizable desktop sidebar and scroll-safe compact/mobile drawer.
• Versioned datasets and immutable test cases with source text, primary URL, expected behaviour, privacy class and tags.
• An integrated run console with a browser-local deterministic runner, optional localhost runner bridge and import-only boundary.
• Append-only harness and configuration versions carrying a workflow profile plus commit or build reference.
• New immutable runs with nine retained artifacts: eight canonical review files plus the original source.
• Automated bundle, provenance, relevance, context, action-specificity and trace evaluators that never populate human scores.
• An operator overview with session progress, a deterministic run queue and a clear next-review action.
• Bundle diagnostics for canonical artifact presence, parse failures, identity consistency, unique IDs and resolvable claim/evidence/action/uncertainty links.
• Multiple agent-agnostic workspaces with editable project, agent and reviewer identities.
• Dated sessions and run groups for repeat reviews over time.
• Folder and loose-file import.
• A complete artifact browser with filtering, role detection, parse state, table preview, image/PDF preview where the browser supports it, raw text/JSON preview and download-copy actions.
• A canonical operator-review order for human packet, claims, evidence, source card, actions, memory, uncertainty and run log, with unknown files retained afterwards.
• Canonical review mapping for human packet, claims, evidence, source, actions, memory, uncertainty and trace artifacts.
• Retention of unfamiliar supporting files instead of silent discard.
• Paired claim/evidence review with surrounding context, a primary-source link and exact local source-line navigation.
• Per-claim categorical judgments including Supported and relevant, Not a claim, Irrelevant, Missing context, Unsupported, Misclassified, Duplicate, Needs external verification and Linked action is unrelated.
• Required corrections for claim-level issue labels, kept separate from five once-per-run product-quality scores.
• A full execution-trace tree alongside action, memory and uncertainty review.
• Explicit operator confirmation for the review contract, source card, every classified claim/evidence pair, actions, memory, uncertainty and run log before completion.
• Five once-per-run quality ratings on a 0–3 scale, a final decision and correction notes.
• A complete rating rubric with dimension-specific 0–3 anchors; correction notes are required for Needs revision and Fail.
• Resumable drafts.
• Immutable review revisions, JSON export, linked re-review and score-change history.
• Session-level evaluation-pack export with diagnostics, latest operator revisions and an explicit completion summary.
• Exact isolation of development-QA seed records from genuine operator progress.
• Baseline-versus-candidate metrics and a separate immutable pairwise preference.
• Aggregate evaluator metrics, failure taxonomy, multi-reviewer agreement and configurable CI regression gates.
• A System inspector that exposes implemented algorithms, data structures, complexity and Year Two computer-science links while marking backpropagation as future work.
• A searchable Agent Harness Field Guide with plain-English terminology, goal-aware evaluation recommendations and a clearly labelled non-executing Python starter template.
• A 19-step guided tour, available from the persistent header information button and Settings, that explains workspace hierarchy and points directly to the real Run button before navigating dataset, lineage, review, comparison, learning and lifecycle controls.
• Persistent light and dark themes for extended evidence reading and focused operator sessions.
• A responsive desktop, tablet and phone layout with contained headers, stacked mobile controls and drawer navigation.
• IndexedDB storage for imported run bundles and binary attachments; local browser storage for identities, drafts and revision records.
• An optional Chaser Agent case-study pack with a relevance-fixed candidate and immutable flawed baseline. It is retained as importable test evidence and is never loaded into a fresh user's workspace list.

File compatibility

The current adapter registry supports:

The detailed adapter and normalization contract is in docs/ARTIFACT_COMPATIBILITY.md.

Run locally

npm install
npm run dev -- --host 127.0.0.1 --port 4173

Open the address printed by Vite. Data remains in the browser profile on this computer.

The browser-local runner needs no second process. To test the bounded localhost bridge in another terminal:

npm run runner

The bridge binds to 127.0.0.1:4318, accepts only localhost Studio origins, creates a new immutable bundle and performs no provider call, external action, memory promotion or model training.

Verification

npm test
npm run build

The test suite covers adapter aliases, JSON Lines recovery, nested run grouping, empty first-run workspace state, legacy personal-workspace migration, workspace purpose/goal persistence, deterministic diagnostics, claim-label contracts, ranked extraction, immutable run creation, goal-aware evaluation recommendations, optional case-study fixtures, automated evaluators, baseline comparison, reviewer agreement, CI gates, revision lineage and Sites packaging. Browser interaction and personal-instance migration proof are recorded in design-qa.md.

Data and safety boundary

• Imported artifacts are read-only evidence.
• Finishing a review creates a new review revision and exports JSON; it never rewrites the imported bundle.
• Re-review links a new revision to its parent instead of overwriting the earlier judgement.
• No provider call, external tool action, memory promotion, training job or deployment is authorized by a review.
• Version 1.5.0 remains local-first. The localhost bridge is a bounded single-operator adapter, not a hosted multi-user execution service. Authentication, queues, rate limiting and network context transport remain a later governed HTTP/server phase.

Open-source release

Version 1.5.0 is licensed under Apache-2.0 and includes contribution and security policies. The source is public at github.com/chasedndt/agent-review-studio, and the working application is deployed through OpenAI Sites. The package remains private: true to prevent accidental npm registry publication; that flag does not limit source-code use under the repository license.

Engineering map

• Artifact compatibility
• Review revision model
• Phase 2 implementation record
• Operator completion implementation record
• Version 1.0 release record
• Version 1.0.2 identity, responsive and Chaser-instance record
• Version 1.1 industry workbench implementation
• Operator guide
• Version 1.2 release-readiness audit
• Version 1.3 navigation, onboarding and lifecycle record
• Version 1.3.1 hierarchy, typography and responsive repair
• Version 1.4 workspace accordion and resizable sidebar
• Version 1.5 agent-agnostic first run
• Chaser Agent case-study and redaction contract
• Documentation index