🧠 AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

AMA-Bench is a benchmark for evaluating agent memory on long-horizon agent trajectories.

📚 Category

• 🎬 Media
• 🗞️ News
• 🧭 Overview
• 🧠 Memory Architecture (Two-Stage)
• ⚙️ Installation
• 📦 Dataset
• 🚀 Quick Start
• 📊 Evaluation
• 🧪 Experiment Log
• 🧩 Scripts
• 🏆 Submit to Leaderboard
• 🏗️ Project Structure
• 🧪 Synthetic Data Generation
• 🛠️ Add a New Method
• 📖 Citation
• 📮 Contact
• 📄 License

🎬 Media

• AMA-Bench was featured by dair.ai: https://x.com/dair_ai/status/2027776582262395054
• Special thanks to Vincent De Genova for the visual notes and homepage feature: https://vdegenova.github.io/visual-notes/visuals/ama-bench/

🗞️ News

• ✅ [2026-05] 🎉 AMA-Bench paper accepted to ICML 2026!
• ✅ [2026-05] 🤖 AMA-Agent code release
• ✅ [2026-03] 🚀 repo launched
• ✅ [2026-03] 📄 AMA-Bench paper accepted to ICLR 2026 Memory Agent workshop — selected for oral presentation 🎤 — and released on arXiv!
• ✅ [2026-03] 📦 Dataset released on Hugging Face
• ✅ [2026-03] 🏆 Leaderboard launched on Hugging Face Spaces

---

🧭 Overview

AMA-Bench evaluates whether a method can:

• Build memory from long agent trajectories
• Retrieve relevant evidence for a given question
• Answer open-ended and MCQ questions robustly

All methods share a unified two-stage interface defined in src/method/base_method.py:

• memory_construction(traj_text, task="") — build a memory object from trajectory text
• memory_retrieve(memory, question) — retrieve relevant context for a question

🧠 Memory Architecture (Two-Stage)

Each memory method is implemented in two stages:

---

⚙️ Installation

Requirements: Python 3.9–3.11 · Linux · CUDA GPU (recommended)

git clone https://github.com/AMA-Bench/AMA-Hub.git
cd AMA-Hub
python -m venv ama_venv
source ama_venv/bin/activate
pip install -r requirements.txt

---

📦 Dataset

Official dataset: huggingface.co/datasets/AMA-bench/AMA-bench

huggingface-cli download AMA-bench/AMA-bench --repo-type dataset --local-dir ./dataset

The default evaluation path is dataset/test/. The dataset contains only a test split — there is no training split or qa_set directory.

dataset/
└── test/
    └── open_end_qa_set.jsonl

---

🚀 Quick Start

▶️ Run end-to-end evaluation with local vLLM

bash scripts/run.sh

Launches a local vLLM server (per scripts/launch_vllm_32B.sh) and then runs generation + LLM-as-judge. All logs (vLLM server + pipeline output) are written to logs/<timestamp>/.

☁️ Run end-to-end evaluation against an API

bash scripts/run_api.sh

Skips the vLLM launch and calls a remote API (OpenAI / Gemini / Anthropic, etc.) configured via LLM_CONFIG. Pipeline output is written to logs/<timestamp>/run.log.

💻 Run via CLI

python src/run.py \
  --llm-server vllm \
  --llm-config configs/qwen3-32B.yaml \
  --subset openend \
  --method longcontext \
  --test-dir dataset/test \
  --judge-config configs/llm_judge.yaml

🤖 Run a tool-using agent (codex / claude_code)

You can also evaluate an external coding agent that reads the trajectory with its own tools (grep/jq/python over many turns) and answers a whole episode in one session:

python src/run.py \
  --llm-server api \
  --llm-config configs/codex_gpt5_mini.yaml \
  --judge-config configs/llm_judge_gpt5_mini.yaml \
  --subset openend \
  --method codex \
  --episode-ids 30,31,32,33,34,35,36,37,38,39
# swap in --method claude_code to run Claude Code instead

The agent gets the full, untruncated trajectory as trajectory.jsonl. codex uses an OpenAI-compatible key (OPENAI_API_KEY, optional OPENAI_BASE_URL); claude_code uses your local Claude Code login (or the standard ANTHROPIC_* env).

---

📊 Evaluation

src/run.py runs answer generation followed by LLM-as-judge scoring, saving outputs to results/.

To evaluate an existing answer file separately:

bash scripts/evaluate.sh \
  --answers-file results/openend/answers_xxx.jsonl \
  --test-file dataset/test/open_end_qa_set.jsonl \
  --judge-config configs/llm_judge.yaml

Judge Reliability

We performed a comprehensive consistency check for the LLM-as-judge setup, evaluating multiple judge models and analyzing their agreement patterns.

Performance against human annotations

Overall Judge Accuracy Scores

Model Agreement Rates

Highest Agreement (> 90%):

• GPT-5.2 vs. GPT-5.4: 94.58% ✅ (Highest agreement - same family)
• DeepSeek-v3.2 vs. GPT-5.2: 90.20% ✅
• Claude-4.6 vs. GPT-5.2: 90.16% ✅

Qwen3-32B Agreement with Others:

• vs. GPT-5.4: 92.80%
• vs. GPT-5.2: 91.80%
• vs. Claude-4.6: 88.80%
• vs. DeepSeek-v3.2: 84.70%

Leniency Bias Analysis

Qwen3-32B shows systematic leniency bias, particularly on:

• SOFTWARE domain (swebench): 100 lenient cases vs. 15 strict (vs. Claude-4.6)
• Game domain: 66 lenient cases vs. 17 strict
• TEXT2SQL domain (spider2): 62 lenient cases vs. 24 strict

Example 1: Accepting Core Concepts Without Complete Details

Task: babaisai (Game domain) Question: Based on game mechanics, what hidden movement mechanic can be inferred when pushing blocks?

Reference Answer: When pushing a text block, that block and adjacent blocks forming a phrase move as a group diagonally (one step in push direction and one step to the left).

Predicted Answer: Pushing a block causes adjacent blocks in the same rule line to shift, suggesting rule blocks are connected and move together.

• Qwen3-32B: ✅ CORRECT (accepts answer capturing the core mechanic despite missing diagonal movement detail)
• Claude-4.6: ❌ INCORRECT (requires the specific diagonal movement detail)

Example 2: Accepting Partial Explanations

Task: crafter (Game domain) Question: What inventory change occurred and why is it critical for future crafting?

Reference Answer: Agent gained wood. Critical because wood is needed to craft a workbench, and without a workbench, the agent cannot craft any other tools, halting technological progression.

Predicted Answer: Agent gained wood. Critical because wood is required to craft tools (e.g., Wood Pickaxe), which are prerequisites for mining stone and other resources.

• Qwen3-32B: ✅ CORRECT (accepts the wood→tools explanation despite skipping the intermediate workbench step)
• Claude-4.6: ❌ INCORRECT (requires explicit mention of the workbench as the critical intermediate step)

Recommendations: While Qwen3-32B achieves the highest accuracy, its leniency bias means it may accept incomplete answers. For critical evaluations, consider using Claude-4.6 as the reference judge or employing ensemble voting across multiple judges.

---

🧪 Experiment Log

Full per-run logs (model outputs, judge outputs, and pipeline traces) for our AMA-Bench / AMA-Agent experiments are mirrored to Google Drive:

📂 AMA-Bench / AMA-Agent experiment logs

Use these to reproduce reported numbers, inspect raw judge decisions, or compare your own runs against ours.

---

🧩 Scripts

launch_vllm_32B.sh — reads model and server settings from YAML, starts a vLLM OpenAI-compatible API, waits for the health endpoint. The vLLM log path is controlled by VLLM_LOG (default logs/vllm_server.log).

run.sh — local-vLLM pipeline. Launches the vLLM server if needed, runs answer generation with the configured method, then evaluates with LLM-as-judge. Per-run logs (including the vLLM server log) are written to logs/<RUN_ID>/.

run_api.sh — API-based pipeline. Same generation + judge flow as run.sh but skips the vLLM launch and points --llm-server at api. Per-run logs are written to logs/<RUN_ID>/.

evaluate.sh — standalone judge evaluation for pre-generated answer files, useful for re-evaluating with a different judge model.

run_cross_validation.sh — evaluates answers with both GPT-5.2 and Qwen3-32B judges, then compares results to measure inter-rater reliability.

cross_validate.py — compares two evaluation result files and generates agreement statistics, including Cohen's Kappa and detailed disagreement analysis.

---

🏆 Submit to Leaderboard

Visit the AMA-Bench Leaderboard and upload a JSONL file where each line is:

{"episode_id": 0, "answer_list": ["(A)", "(B)(C)"], "reasoning_trace": "..."}

Required fields: episode_id, answer_list. reasoning_trace is optional.

---

📮 Contact

• yuz285@ucsd.edu

---

🏗️ Project Structure

AMA-Bench/
├── src/
│   ├── method/             # Memory method implementations (BM25, embedding, AMA-Agent, …)
│   ├── synthetic_data_gen/ # Trajectory synthesis tools (BabyAI, TextWorld, Crafter)
│   ├── run.py              # Main evaluation entry point
│   ├── evaluate.py         # LLM-as-judge evaluation
│   └── model_client.py     # Unified client for different LLM providers
├── configs/                # Model / judge / method configs
│   ├── llm_judge.yaml      # VLLM judge config (Qwen3-32B)
│   └── llm_judge_api.yaml  # API judge config (GPT-5.2, etc.)
├── scripts/                # Shell scripts for launching and evaluation
│   ├── run.sh
│   ├── evaluate.sh
│   ├── run_cross_validation.sh  # Cross-validation pipeline
│   └── cross_validate.py        # Cross-validation comparison tool
├── docs/                   # Documentation
│   └── CROSS_VALIDATION.md      # Cross-validation guide
├── dataset/                # Local dataset (test split, downloaded separately)
├── results/                # Prediction outputs and evaluation metrics
├── assets/                 # Figures
└── utils/                  # Shared utilities

🧪 Synthetic Data Generation

src/synthetic_data_gen/ provides tools for generating trajectories used in the benchmark:

• BabyAI — grid-world navigation tasks with natural language instructions
• TextWorld — text-based interactive fiction environments
• Crafter — pixel-art 2D open-world survival environment

Both include trajectory generation, automatic QA pair creation via state tracking, and token-based length binning. See src/synthetic_data_gen/README.md for details.

---

🛠️ Add a New Method

Create a subclass of BaseMethod and implement both stages:

from src.method.base_method import BaseMethod

class MyMethod(BaseMethod):
    def memory_construction(self, traj_text: str, task: str = ""):
        # Build and return a memory object from the trajectory
        ...

    def memory_retrieve(self, memory, question: str) -> str:
        # Return relevant context string for the question
        ...

Then register it in src/method_register.py to make it available via --method.

Add a tool-using agent harness

If your method is an external agent that reads the trajectory with its own tools (instead of the two-stage build/retrieve flow), subclass AgentHarnessMethod. It is driven once per episode with the raw trajectory and all questions, and returns the Answer[i] block:

from src.method.agent_method import AgentHarnessMethod

class MyAgentMethod(AgentHarnessMethod):
    def run_episode(self, *, trajectory, task, questions, mcq_mode) -> str:
        # run your agent over `trajectory`, return the "Answer[i]: ..." block
        ...

Register it the same way ("my_agent": ("src.method.agent_method", "MyAgentMethod")) and --method my_agent just works — no change to the eval loop. See codex / claude_code for working examples.

---

📖 Citation

@misc{zhao2026amabenchevaluatinglonghorizonmemory,
  title={AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications},
  author={Yujie Zhao and Boqin Yuan and Junbo Huang and Haocheng Yuan and Zhongming Yu and Haozhou Xu and Lanxiang Hu and Abhilash Shankarampeta and Zimeng Huang and Wentao Ni and Yuandong Tian and Jishen Zhao},
  year={2026},
  eprint={2602.22769},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2602.22769}
}

📄 License

MIT License. See LICENSE.