AgentRx Trace Demo

AgentRx Trace Demo is an interactive debugging and reporting interface for analyzing agent trajectories end to end. It combines a Python backend pipeline with a React dashboard so you can:

• upload a trajectory file
• normalize it into a canonical IR
• run invariant generation and checking
• classify the likely root-cause failure with an LLM judge
• inspect the full interaction trace
• review generated plots, metrics, token usage, and estimated cost
• export run metrics as CSV and download generated plots

The current UI is branded as an OpenAI-backed demo and is designed for internal demos, evaluation workflows, and trace forensics.

Highlights

• Real backend execution from the UI via Run Demo
• Support for multiple trajectory formats, including flash, tau-retail, and magentic-one
• End-to-end trace playback with step inspection and modal zoom
• Root-cause categorization and invariant violation analysis
• Reports dashboard with:
  • predicted failure counts
  • root cause step distribution
  • checker pass rate
  • input, output, and total tokens
  • estimated cost
  • downloadable plot assets
  • CSV export for Excel

Repository Structure

AgentRx/
├─ backend/
│  ├─ run.py                      # Main pipeline runner
│  ├─ reports_server.py           # Lightweight local reports/demo server
│  ├─ requirements.txt           # Python dependencies
│  ├─ src/
│  │  ├─ ir/                     # IR normalization
│  │  ├─ invariants/             # Static + dynamic invariants
│  │  ├─ judge/                  # LLM-as-a-Judge
│  │  └─ reports/                # Reporting API layer
│  ├─ trajectories/              # Example input trajectories
│  ├─ uploads/                   # Uploaded files from UI runs
│  └─ runs/                      # Generated run artifacts
├─ frontend/
│  ├─ src/                       # React app
│  ├─ public/
│  └─ package.json
└─ README.md

Recommended Environment

The project is currently best aligned with the following setup:

• Python: 3.11
• Conda environment: agentrx
• Node.js: 20+
• npm: 10+
• OS: Windows is the current validated path for this repo

The backend in your current setup is already using:

• Python 3.11.15
• Conda env agentrx

Prerequisites

Before starting, make sure you have:

• Anaconda or Miniconda installed
• Node.js installed
• Access to either the OpenAI API or Azure OpenAI
• A valid API key for whichever provider you choose

Installation

1. Clone the repository

git clone <your-repo-url>
cd AgentRx

2. Create and activate the conda environment

If you have not already created the environment:

conda create -n agentrx python=3.11 -y
conda activate agentrx

3. Install backend dependencies

cd D:\AgentRx
python -m pip install -r backend\requirements.txt

4. Install frontend dependencies

cd D:\AgentRx\frontend
npm install

Backend Configuration

The frontend lets you choose the provider, but it does not collect secrets. Runtime keys and endpoints are taken from the backend .env file only.

Create:

D:\AgentRx\backend\.env

Example:

# Provider selected by default when running from CLI.
# The UI can override this per run with the provider dropdown.
AGENT_VERIFY_LLM_PROVIDER=openai

# OpenAI
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-5.4-nano
OPENAI_RATE_LIMIT_TIER=tier1

# Optional: custom OpenAI-compatible base URL
# OPENAI_BASE_URL=https://api.openai.com/v1

# Azure OpenAI, only needed if you choose Azure OpenAI in the UI
AGENT_VERIFY_ENDPOINT=https://your-resource-name.openai.azure.com/
AGENT_VERIFY_DEPLOYMENT=your_azure_deployment_name
AGENT_VERIFY_API_VERSION=2025-04-01-preview
AGENT_VERIFY_API_KEY=your_azure_openai_api_key_here

# Optional: token-cost reporting in the reports dashboard
AGENT_VERIFY_INPUT_COST_PER_1K=0.0002
AGENT_VERIFY_OUTPUT_COST_PER_1K=0.00125

Notes

• OPENAI_API_KEY is required for real pipeline runs
• OPENAI_MODEL controls the model used when provider is OpenAI
• OPENAI_RATE_LIMIT_TIER controls GPT-5.4 nano TPM/RPM utilization calculations in the Reports tab
• OPENAI_BASE_URL is optional and only needed for an OpenAI-compatible gateway
• AGENT_VERIFY_ENDPOINT, AGENT_VERIFY_DEPLOYMENT, AGENT_VERIFY_API_VERSION, and AGENT_VERIFY_API_KEY are required when provider is Azure OpenAI
• For Azure OpenAI, AGENT_VERIFY_ENDPOINT should be the base resource URL only, not a full /openai/... route
• Cost values are optional but recommended if you want the Estimated Cost card to populate

GPT-5.4 Nano Rate Limits

For OpenAI gpt-5.4-nano, the Reports tab can calculate run utilization against the selected usage tier. Set OPENAI_RATE_LIMIT_TIER in backend/.env to one of tier1, tier2, tier3, tier4, or tier5.

Running the Application

You need two terminals.

Terminal 1: Start the backend reports server

conda activate agentrx
cd D:\AgentRx
python backend\reports_server.py

The server runs locally at:

http://127.0.0.1:8000

Terminal 2: Start the frontend

conda activate agentrx
cd D:\AgentRx\frontend
npm run dev

Open the Vite URL shown in the terminal, typically:

http://localhost:5173

How to Use

1. Upload a trajectory file

You can start with any of the included examples:

• backend/trajectories/test_random_format.json
• backend/trajectories/tau-retail/instruction_adherence_failure.json
• backend/trajectories/tau-retail/underspecified_intent.json
• backend/trajectories/magentic-one/trajectories/intent_not_supported.json

2. Click Run Demo

Choose either OpenAI or Azure OpenAI from the provider dropdown, then click Run Demo.

This triggers the real backend pipeline:

1. IR normalization
2. Static invariant generation
3. Dynamic invariant generation
4. Invariant checking
5. Judge classification
6. Report generation

3. Review the tabs

• Trace: end-to-end interaction trace
• Step Detail: expanded view of a selected step
• Judge Output: root-cause output and violations
• Raw JSON: normalized/raw trajectory content
• Reports: plots, metrics, token usage, cost, and CSV export

Supported Inputs

The backend supports multiple trajectory shapes and can auto-detect or normalize them.

Currently included examples cover:

• flash
• tau-retail
• magentic-one

If a domain-specific converter produces weak IR, the backend can fall back to an LLM-based IR normalization path.

Reports and Exports

The Reports tab includes:

• available runs
• predicted failure categories
• root-cause step positions
• checker assertion outcomes
• run tasks
• generated plots
• token utilization
• estimated cost

Export options

• Download: downloads plot files such as predicted.png
• Export CSV: exports the selected run’s metrics in Excel-friendly CSV format

Manual CLI Pipeline Usage

You can also run the pipeline directly without the UI.

From D:\AgentRx\backend:

conda activate agentrx
cd D:\AgentRx\backend
python run.py trajectories\test_random_format.json

Examples:

python run.py trajectories\test_random_format.json --stage ir
python run.py trajectories\test_random_format.json --skip-judge
python run.py trajectories\tau-retail\instruction_adherence_failure.json --domain tau
python run.py trajectories\test_random_format.json --endpoint openai
python run.py trajectories\test_random_format.json --endpoint azure

Run artifacts are written under:

D:\AgentRx\backend\runs\<run_name>

Troubleshooting

No module named 'openai'

You are likely not running inside the agentrx conda environment.

Check:

conda activate agentrx
python -c "import sys; print(sys.executable)"
python -c "import openai; print(openai.__version__)"

OpenAI API key issues

For OpenAI runs, set your key in D:\AgentRx\backend\.env:

OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-5.4-nano
OPENAI_RATE_LIMIT_TIER=tier1

For Azure OpenAI runs, set:

AGENT_VERIFY_ENDPOINT=https://your-resource-name.openai.azure.com/
AGENT_VERIFY_DEPLOYMENT=your_azure_deployment_name
AGENT_VERIFY_API_VERSION=2025-04-01-preview
AGENT_VERIFY_API_KEY=your_azure_openai_api_key_here

Frontend looks stale after code changes

Restart Vite:

cd D:\AgentRx\frontend
npm run dev

Then hard-refresh the browser.

Security Notes

The current repo is suitable for local/internal demo use. Some protections have already been added:

• request upload size limit
• local-origin CORS restriction
• concurrent demo job cap
• prompt hardening for untrusted trajectory content
• safer handling around missing judge output

Still recommended before wider deployment:

• add authentication
• add stronger rate limiting
• add stricter schema validation for uploads
• fully remove legacy code paths that are no longer used

Hardening Status

The table below summarizes what has already been tested and fixed, and what is still recommended before taking the project further toward production.

Current Productionization Priorities

If you want to continue hardening the project, the next best steps are:

1. Add authentication and stronger rate limiting to the backend server.
2. Tighten uploaded trajectory validation with an explicit schema and safer preprocessing.
3. Remove legacy code paths that are no longer allowed in the runtime.
4. Run end-to-end browser smoke tests and HTTP-level load tests outside the current sandboxed environment.

Ownership

Created by Vijay Krishnan MR
Contact: vijaykrishnanmr@gmail.com

See LICENSE for repository usage terms.