Interactive Evaluation Requires a Design Science

If you find this work useful, please consider reading our full paper and citing:

@misc{xuan2026interactiveevaluationrequiresdesign,
      title={Interactive Evaluation Requires a Design Science}, 
      author={Keyang Xuan and Peiyang Song and Pan Lu and Pengrui Han and Wenkai Li and Zhenyu Zhang and Zexue He and Wenyue Hua and Manling Li and Jiaxuan You and Adrian Weller and Yizhong Wang and Jiaxin Pei},
      year={2026},
      eprint={2605.17829},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.17829}, 
}

---

💡 Core Insight

Interactive evaluation should be built as a design science for evaluating systems acting through trajectories. The field does not merely need more interactive benchmarks; it needs explicit principles for specifying what interaction artifacts enter evaluation and how an evaluation program maps those artifacts to judgments.

Response-centered evaluation remains useful when final outputs are sufficient evidence.
Interactive evaluation becomes necessary when systems act through tools, environments, users, or other agents, so that earlier actions shape later evidence, opportunities, risks, and outcomes.
The central design question is what trajectory evidence enters evaluation and how that evidence is mapped to system-level judgments.

🎯 The Framework

We define evaluation as an autonomous mapping:

E : X → Y

Where:

• X = Admissible evidence (expands from final responses to interaction-generated trajectories)
• E = Evaluation program (maps trajectories to judgments about system-level performance)

What Changes in Interactive Evaluation?

🗺️ 2D Taxonomy of Interactive Evaluation

We propose organizing interactive evaluation along two orthogonal axes:

Axis 1: Evaluation Inputs (What trajectories connect to)

• 🛠️ Tools & Environments — Web pages, OS, apps, repositories
• 👤 Users — Human feedback, clarification, evolving instructions
• 🤝 Other Agents — Coordination, negotiation, multi-agent systems
• 🌐 Hybrid & Dynamic — Persistent state, cross-session dependencies

Axis 2: Evaluation Programs (How trajectories → judgments)

• ✅ Task Success — Final goal completion
• ⚙️ Process Quality & Efficiency — Tool choice, action economy, code locality
• 🔄 Recoverability & Robustness — Error detection, plan revision, adaptation
• 🛡️ Safety, Alignment & Social Competence — Norm-sensitive behavior, cooperation

Key Findings:

• 🎯 Trajectory evidence remains outcome-centered — many benchmarks record trajectories but only score final success
• 🏗️ Evaluation programs are substrate-bound — metrics follow what's easy to measure, not what claims require
• ⚠️ Hybrid & dynamic systems severely underexplored — critical gap as systems move toward longer horizons

📐 Design Principles

Our framework proposes five core principles for interactive evaluation:

1. 📝 Specify the System and Trajectory Evidence

Clarify what system is evaluated, what resources it accesses, and what claims the trajectory supports.

2. 🔧 Specify the Interaction Protocol

Document initial state, allowed actions, observation space, stopping rules, reset conditions — the "dataset documentation" of interactive evaluation.

3. 🔄 Design for Perturbation and Repair

Test whether systems detect problems, revise strategies, and remain effective under changing conditions.

4. 📊 Separate Outcome, Process, and Risk

Report final success, trajectory-level properties (cost, safety, recovery), and risks separately.

5. 🏗️ Build Shared Infrastructure Without Freezing Design

Create reusable environments, logging schemas, and reporting templates while preserving diversity in protocols.

---

📊 Representative Benchmarks (Kept Updating)

We curated and categorized 56 benchmarks across three evolutionary stages:

Stage 1: Response-Centered

Stage 2: Task-Driven

Stage 3: Interactive

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

📬 Contact

For questions or collaborations, contact:

• Keyang Xuan: keyangx@utexas.edu
• Peiyang Song: psong2@andrew.cmu.edu

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgment

• We thank Daniel Fried (CMU) and Katie M. Collins (Cambridge/MIT/Princeton) for discussions and helpful feedback on earlier versions of this paper.

📖 Citation

If you find our work useful, please consider citing our paper:

@misc{xuan2026interactiveevaluationrequiresdesign,
      title={Interactive Evaluation Requires a Design Science}, 
      author={Keyang Xuan and Peiyang Song and Pan Lu and Pengrui Han and Wenkai Li and Zhenyu Zhang and Zexue He and Wenyue Hua and Manling Li and Jiaxuan You and Adrian Weller and Yizhong Wang and Jiaxin Pei},
      year={2026},
      eprint={2605.17829},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.17829}, 
}

---