📝 Abstract

Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks, but it remains unclear whether such experience can be distilled into reusable procedural skills. We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation. It contains 180 tasks across six real-world agent environments, organized into role-conditioned task families with shared latent procedures. Agents learn from acquisition tasks, update an external skill library using compacted trajectories and verifier feedback, and then face frozen deployment tasks testing context shift, adversarial shortcuts, and composition.

By comparing self-generated and curated-start skill evolution against no-skill and raw-trajectory controls, SkillEvolBench separates procedural abstraction from base capability, curated prior knowledge, and direct reuse of episodic traces. Across ten model configurations and three agent harnesses, we find that current agents often adapt locally but rarely form robust reusable skills. Skill-based conditions can improve acquisition or replay, and individual models sometimes gain on specific deployment axes, but these gains are unstable under frozen deployment.

Raw-trajectory reuse frequently outperforms distilled skills, suggesting that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. Capacity and cost analyses further show that writing more skills or larger Tier-3 resource libraries is not sufficient: additional updates can improve coverage while introducing episode-specific drift and procedural clutter. These findings position SkillEvolBench as a testbed for measuring when one-off experience becomes durable procedural knowledge rather than task-local memory.

🗂️ GitHub Repo Layout

🧩 Benchmark Structure

SkillEvolBench uses a fixed stratified design:

6 environments × 5 latent skill families × 6 tasks = 180 tasks

For each latent skill family, T1-T3 are learning/acquisition tasks and T4-T6 are frozen evaluation tasks:

⚙️ Installation

Python 3.12 is recommended.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'
python -m pip install litellm boto3 jinja2 pandas matplotlib

Install Harbor and build the task runtime image:

python -m pip install git+https://github.com/harbor-framework/harbor.git
bash docker/agent-build/build.sh
docker image inspect agent-runtime:latest

🔐 API Keys and Provider Setup

Create a local credential file. It is ignored by Git.

touch .harbor-agents.env
chmod 600 .harbor-agents.env

Fill only the providers you plan to run:

# Azure OpenAI / Codex presets
AZURE_OPENAI_ENDPOINT=https://<your-resource>.openai.azure.com/openai/v1
AZURE_OPENAI_API_KEY=<your-azure-openai-key>
# Optional if your endpoint requires it:
# AZURE_OPENAI_API_VERSION=<api-version>

# AWS Bedrock Claude presets
AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-bearer-token>
AWS_REGION=us-east-1
CLAUDE_CODE_USE_BEDROCK=1

# Gemini direct presets
GEMINI_API_KEY=<your-gemini-api-key>

# Kimi / OpenAI-compatible Mantle endpoint
KIMI_BEDROCK_BASE_URL=<your-openai-compatible-base-url>
KIMI_BEDROCK_API_KEY=<your-kimi-key>

Load credentials in the same shell before launching runs:

set -a
source .harbor-agents.env
set +a

Check without printing secret values:

python -c "import os; [print(k, bool(os.getenv(k))) for k in ['AZURE_OPENAI_ENDPOINT','AZURE_OPENAI_API_KEY','AWS_BEARER_TOKEN_BEDROCK','AWS_REGION','GEMINI_API_KEY','KIMI_BEDROCK_BASE_URL','KIMI_BEDROCK_API_KEY']]"

Provider notes

The paper configuration here uses Azure OpenAI for Codex-style models and AWS Bedrock Claude for Claude. If you use direct OpenAI/Anthropic API keys instead, add a new YAML under configs/models/ and set the right provider, harbor_agent_name, agent_model_name, agent_env, and host_litellm fields for your endpoint.

✅ Validate Before Running

python -m scripts.validate_configs
python -m scripts.validate_assets
python -m scripts.preflight
python -m scripts.run --baseline-name no_skill --order-seed A --dry-run

A healthy dry run schedules 180 tasks and does not call any model API.

For a stricter check including Harbor and Docker image readiness:

python -m scripts.preflight --strict

🚀 Run One Benchmark

General pattern:

python -m scripts.run \
  --baseline-name <baseline_name> \
  --model-yaml configs/models/<model_preset>.yaml \
  --order-seed A

Azure OpenAI example:

python -m scripts.run \
  --baseline-name no_skill \
  --model-yaml configs/models/gpt-5.4.yaml \
  --order-seed A

AWS Bedrock Claude example:

python -m scripts.run \
  --baseline-name selfgen_experience_always \
  --model-yaml configs/models/claude-sonnet-4.6.yaml \
  --order-seed A

Use --run-id for a stable output directory:

python -m scripts.run \
  --baseline-name raw_trajectory_rag \
  --model-yaml configs/models/gpt-5.4.yaml \
  --order-seed A \
  --run-id rawrag_gpt54_seedA

📊 Paper Baselines

Canonical baselines are defined in skillevolbench/baselines/policy.py.

Run the full canonical ladder for one model:

MODEL=configs/models/gpt-5.4.yaml
BASELINES=(
  no_skill
  raw_trajectory_rag
  selfgen_zero_shot
  selfgen_experience_always
  curated_static
  curated_with_revision_always
)

for BASELINE in "${BASELINES[@]}"; do
  python -m scripts.run \
    --baseline-name "$BASELINE" \
    --model-yaml "$MODEL" \
    --order-seed A
done

Or use the multi-run launcher after inspecting the plan:

python -m scripts.launch_multi_model \
  --baselines no_skill,raw_trajectory_rag,selfgen_zero_shot,selfgen_experience_always,curated_static,curated_with_revision_always \
  --models gpt-5.4 \
  --order-seed A \
  --max-workers 1 \
  --dry-run

Remove --dry-run to execute.

🧪 Ablation Baselines

Run an ablation like any other baseline:

python -m scripts.run \
  --baseline-name selfgen_experience \
  --model-yaml configs/models/gpt-5.4.yaml \
  --order-seed A

🤖 Model Presets

Inspect a preset before editing:

sed -n '1,120p' configs/models/gpt-5.4.yaml

📦 Outputs

Runs are written to:

workspace/runs/<run_id>/

Key files:

Summarize a run:

python -m scripts.summarize workspace/runs/<run_id>

Find recent runs:

ls -lt workspace/runs | head

🛠️ Troubleshooting

📚 Citation

@article{lei2026skillevolbench,
  title={SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills},
  author={Lei, Yingtie and Wan, Zhongwei and Zhang, Jiankun and Alam, Samiul and Zhong, Zixuan and Huang, Peizhou and Wang, Xin and Zhang, Jingxuan and Zhou, Donghao and Hsieh, Yunta and others},
  journal={arXiv preprint arXiv:2605.24117},
  year={2026}
}