Vestige: Code Agent Trajectory Analysis Tool

Vestige is the companion tool for the paper "On Randomness in Agentic Evals". It analyzes code agent trajectories on software engineering benchmarks like SWE-Bench, enabling researchers to inspect agent behavior, understand failure modes, and generate publication-ready visualizations and statistical summaries.

Paper

This tool was developed to support the analysis presented in:

Bjarnason, B. H., Silva, A., & Monperrus, M. (2026). On Randomness in Agentic Evals. arXiv preprint arXiv:2602.07150.

See Citing for the BibTeX entry.

Features

• Multi-format trajectory loading: Supports nano-agent, r2e-gym, and Claude Code trajectory formats
• Variance analysis: Computes pass@k and pass^k metrics across multiple runs
• Non-determinism detection: Identifies divergence across temperature=0 runs
• Comparative analysis: Cross-run comparisons, fine-tuning impact evaluation
• Publication-ready outputs: High-quality plots and Markdown tables suitable for papers

Installation

Requires Python >= 3.11.

pip install -e .

Quick Start

Vestige uses Hydra for configuration. Create a YAML config file and run:

vestige --config-path conf --config-name my_config

Example configs are provided in src/vestige/conf/.

Configuration

A run configuration looks like this:

output_dir: "./plots"
max_k: 5

plots:
  - comparison
  - variance
  - non_determinism

# Optional: load SWE-Bench instances for difficulty metrics
instances:
  source: SWE-bench/SWE-bench_Verified
  split: test

runs:
  - name: "my-run"
    format: "nano_agent"          # nano_agent | r2e_gym | claude_code
    trajectories: "/path/to/trajectories.jsonl"
    results: "/path/to/results.json"
    base_model: "Qwen/Qwen3-32B"
    scaffold: "nano-agent"
    temperature: 0.6

Supported Formats

Glob patterns are supported for trajectories paths.

Output

Plots are saved under output_dir/ organized by analysis type:

output_dir/
├── comparison/        # Cross-run comparison plots
├── variance/          # pass@k / pass^k plots
├── non_determinism/   # Divergence analysis plots
└── swebench_scores_table.md

Project Structure

src/vestige/
├── __main__.py          # CLI entry point
├── analysis/            # Metrics, comparison, variance, non-determinism
├── loaders/             # Format-specific trajectory parsers
├── models/              # Trajectory, Run, and Instance data models
├── plotting/            # Visualization modules
└── conf/                # Example Hydra configuration files

Citing

If you use Vestige in your research, please cite:

@article{bjarnason2026randomness,
  title={On Randomness in Agentic Evals},
  author={Bjarnason, Bjarni Haukur and Silva, Andr{\'e} and Monperrus, Martin},
  journal={arXiv preprint arXiv:2602.07150},
  year={2026}
}