🚀 Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems

---

🥳 News

• [2026-10-09] Skill-MAS now supports Codex and Claude Code! After installation, it automatically selects the appropriate optimized skill for the current task and orchestrates the workflow alongside your existing agent skills. Quick start.
• [2026-08-21] 🎉 Skill-MAS has been accepted to EMNLP 2026!
• [2026-06-17] We release the code and paper: Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems.

---

Table of Contents

• 1. Overview
• Use in Codex or Claude Code
• 2. Setup
• 3. Demo Inference (Custom Question)
• 4. Running Evolution
• 5. Evaluating a Fixed Skill
• 6. Tips
• 7. BibTeX

---

1. Overview

Skill-MAS evolves a single meta-agent skill file (SKILL.md) that instructs an LLM to design and orchestrate a Multi-Agent System (MAS) in three stages:

Trajectories from each evolution round drive contrastive reflection and skill rewriting; the best skill snapshot is selected across rounds.

Three ways to use this repo

---

Repository Layout

Skill_MAS/
├── assert/               # README figures (background.png, Method.png)
├── core/                 # CLI entry, evolution pipeline, resume, task selection
├── evolution/            # Rollout, contrastive reflection, skill optimizer, bench adapters
├── skill_mas/            # 3-stage MAS builder, async LLM client, model_config.json
├── template/             # Generated MAS code templates and SubAgent runtime
├── utils/                # Paths, cost tracking, logging, redaction
├── init_skill/           # Initial (pre-evolution) SKILL.md
├── optimized_skill/      # Pre-evolved skills per benchmark (ready to test)
├── skills/skill-mas/      # Portable agent skill + four adapted task profiles
├── scripts/              # Skill installer + default startup routing
├── .claude-plugin/       # Claude Code plugin and marketplace manifests
├── hooks/                # Claude plugin session-start routing
├── docs/agent-skills.md   # Installation, profiles, updates, and adaptation notes
├── dataset/              # Bundled benchmark code + data
│   ├── vitabench/        # VitaBench (src/vita, data/)
│   ├── deep_research_bench/
│   ├── hlemath/
│   └── BrowseComp-Plus/
├── run_*.sh              # Evolution wrappers (run from parent of Skill_MAS/)
├── demo_inference.py     # Single-question build + inference demo
├── demo_inference.sh     # Shell wrapper for demo_inference.py
└── results/              # Generated at runtime (not shipped)

How an evolution round works

1. Multi-trajectory rollout (evolution/rollout_multi.py) — run k trajectories per validation task; the agent reads the current round's SKILL.md, generates MAS code via three build stages (skill_mas/build.py), executes it, and records scores plus phase-level traces.

2. Contrastive reflection (evolution/contrastive_reflect.py) — compare high- vs low-scoring trajectories and synthesize structured improvement signals.

3. Skill bank optimization (evolution/bank_optimizer.py) — an optimizer LLM rewrites the single SKILL.md using reflection reports and round statistics.

4. Round selection (evolution/assemble_select.py) — track per-round scores; after all rounds, select the best skill snapshot.

Benchmark backends

Where results land

Skill_MAS/results/{backend}_{model_tag}/
├── artifacts/
│   ├── skills/{bench_id}/{run_id}/round_XX/SKILL.md   # skill snapshots per round
│   └── runs/{bench_id}/{run_id}/summary_rXX.json      # round metrics
└── log/{bench_id}/{run_id}/round_XX/                  # traces, exports

Each round also saves:

round_r/
  ├── trajectories/     # per-task, per-trajectory records
  ├── aspects/          # phase-level snapshots
  └── contrastive/      # reflection reports

skills/.../round_r/
  ├── SKILL.md          # skill used in this round (rewritten after r-1)
  ├── bank_meta.json    # optimization history
  └── knee_images/      # task-priority elbow plots

---

Use in Codex or Claude Code

Bring Skill-MAS's task decomposition → agent engineering → workflow orchestration into your everyday agent workflow. After installation, Skill-MAS automatically selects the appropriate optimized skill for the current task and orchestrates the workflow alongside your existing skills. The portable skill-mas adapts optimized_skill/ to native agent tools and coordinates delegation and verification as needed. Simple questions and small edits stay lightweight. No benchmark datasets, Python packages, or separate model API keys are needed; your agent's existing tools and access apply.

Install

Clone the repo and run the installer with Python 3.9+:

git clone --depth 1 https://github.com/linhh29/Skill_MAS.git
cd Skill_MAS

# Install for both agents, or choose --agent codex / --agent claude.
python3 scripts/install_agent_skill.py --agent both

This installs the self-contained skill and enables default startup routing. Codex receives a managed block in its AGENTS.md; Claude Code receives a dedicated rules/skill-mas.md. Existing instructions are preserved and changed instruction files are backed up. See the exact paths and controls. For installation in just one project:

python3 scripts/install_agent_skill.py --agent both --scope project --project /path/to/your-project

Use your agent normally

Open a new Codex or Claude Code session in the target project and describe your task:

Compare these three retrieval approaches and recommend one for our project.
Solve this combinatorics problem and explain the proof: [your problem]
Find the project matching these clues and verify the sources: [your clues]
Implement this feature, integrate the changes, and check that it works.

Skill-MAS selects the relevant strategy internally:

Specialist skills handle their own domains. If an existing orchestration skill already coordinates a task, Skill-MAS adds handoff and verification guidance to that workflow, keeping one plan and avoiding duplicate agent teams.

Claude Code plugin alternative

Instead of the copy installer, run these commands inside Claude Code and complete the installation in its plugin interface:

/plugin marketplace add linhh29/Skill_MAS
/plugin install skill-mas@skill-mas-marketplace

Start a new session and use the same ordinary prompts above. The plugin's SessionStart hook loads the routing guidance so the agent can select an optimized skill for the current task. Existing plugin users should update to the latest version.

Choose either the plugin or the copy installation for Claude Code to avoid duplicate entries. Default routing is implemented through the hosts' instruction and hook mechanisms; Skill-MAS is an installed extension, not an official bundled component. The skill uses available native delegation; when subagents are unavailable, it follows the same workflow sequentially and says so. These are adapted orchestration instructions; benchmark results do not establish their performance in Codex or Claude Code.

See the installation and adaptation guide for manual installation, updates, removal, source mapping, and troubleshooting. The research runtime setup below is only needed for the demo and benchmark workflows.

---

2. Setup

Repository layout

Check out this repo so that Skill_MAS/ lives under a parent directory (e.g. demo/ or arxiv_code/). All benchmarks are vendored under Skill_MAS/dataset/; you do not need sibling copies of vitabench_single/, hlemath/, etc.

Python environment

Install dependencies for Skill-MAS and the benchmark you use (see dataset/vitabench/requirements.txt and each benchmark's README).

conda create -n skill_mas python=3.11 -y
conda activate skill_mas
pip install openai pydantic loguru  # plus benchmark-specific deps

Model & API configuration

Edit skill_mas/model_config.json — model ids, pricing, base_url, and ${OPENAI_API_KEY} placeholders. Shell scripts resolve credentials via environment variables:

---

3. Demo Inference (Custom Question)

Use this when you want to try Skill-MAS on your own task without running multi-round evolution.

We ship ready-to-use skill files — no evolution required:

demo_inference.py loads any of these paths, runs the three-stage build, prints Stage-3 MAS code, executes the generated workflow, and prints the answer.

Supported standalone datasets in the demo: hlemath, drb, bcp. VitaBench needs the full simulator (run_vita.sh or dataset/vitabench/run_skill_mas.sh).

Option A — shell script (recommended)

cd /path/to/parent-of-Skill_MAS

# Usage: bash Skill_MAS/demo_inference.sh <model_id> <skill_path> "<question>"

bash Skill_MAS/demo_inference.sh qwen3.5-plus \
  Skill_MAS/init_skill/SKILL.md \
  "What is 17 + 28? Give the final answer in \\boxed{...} form."

bash Skill_MAS/demo_inference.sh qwen3.5-plus \
  Skill_MAS/optimized_skill/hlemath.md \
  "Find the number of positive integers n such that n^2 + 3n + 2 is divisible by n + 1."

Option B — Python directly

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"
export PYTHONPATH="$(pwd):$(pwd)/Skill_MAS/dataset:$(pwd)/Skill_MAS/dataset/vitabench/src"

python Skill_MAS/demo_inference.py \
  --model qwen3.5-plus \
  --skill Skill_MAS/optimized_skill/drb.md \
  --question "Summarize recent progress in retrieval-augmented generation." \
  --verbose

python Skill_MAS/demo_inference.py \
  --model qwen3.5-plus \
  --skill Skill_MAS/init_skill/SKILL.md \
  --question "Your task prompt here." \
  --dataset hlemath \
  --save-mas-code /tmp/demo_mas.py

CLI flags

BrowseComp-Plus skills (bcp) additionally accept --bcp-index-path, --bcp-retrieval-topk, --bcp-doc-max-tokens, and --bcp-max-retrieval-rounds. The BM25 index under dataset/BrowseComp-Plus/scripts_build_index/indexes/bm25 must be available (see that benchmark's README).

For BrowseComp-Plus, refer to the official repo for data preprocessing. We do not ship a plain-text data file used in our paper, to reduce data contamination risk.

---

4. Running Evolution

Quick start (shell scripts)

Run from the directory above Skill_MAS/. Each run_*.sh takes two positional arguments: agent model id and max concurrency.

cd /path/to/parent-of-Skill_MAS

# VitaBench (cross-domain: delivery, instore, ota)
bash Skill_MAS/run_vita.sh <model_id> <max_concurrency>

# Deep Research Bench
bash Skill_MAS/run_drb.sh <model_id> <max_concurrency>

# HLEMath
bash Skill_MAS/run_hlemath.sh <model_id> <max_concurrency>

# BrowseComp-Plus
bash Skill_MAS/run_bcp.sh <model_id> <max_concurrency>

Default evolution settings in the scripts:

Tweak the variables at the top of each script to change rounds, task limits, evaluator/judge models, etc.

CLI (direct invocation)

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"
export PYTHONPATH="$(pwd):$(pwd)/Skill_MAS/dataset:$(pwd)/Skill_MAS/dataset/vitabench/src"

python -m Skill_MAS evolve \
  --bench-backend vitabench \
  --bench-id skill_mas_agent \
  --run-id exp1 \
  --domain "delivery,instore,ota" \
  --jsonl Skill_MAS/dataset/vitabench/data/vita_validate.json \
  --rounds 10 \
  --k-trajectories 5 \
  --agent-llm <model_id> \
  --user-llm <model_id> \
  --evaluator-llm <evaluator_model_id> \
  --optimizer-llm <model_id> \
  --max-concurrency 16 \
  --max-steps 300 \
  --language chinese

Resume and fresh runs

---

5. Evaluating a Fixed Skill

Run a benchmark with an existing skill — no multi-round evolution. The first argument is a path relative to Skill_MAS/ (e.g. init_skill/SKILL.md or optimized_skill/hlemath.md).

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"

bash Skill_MAS/dataset/hlemath/run_skill_mas.sh init_skill/SKILL.md <model_id> <max_concurrency>
bash Skill_MAS/dataset/deep_research_bench/run_skill_mas.sh optimized_skill/drb.md <model_id> <max_concurrency>
bash Skill_MAS/dataset/BrowseComp-Plus/run_skill_mas.sh optimized_skill/bcp.md <model_id> <max_concurrency>
bash Skill_MAS/dataset/vitabench/run_skill_mas.sh optimized_skill/vitabench.md <model_id> <max_concurrency>

---

6. Tips

• Start with demo_inference.sh on one HLEMath question before launching a full evolution run — it is the fastest way to validate API config and skill quality.
• Use --verbose in demo_inference.py to inspect Stage 1–2 JSON; Stage 3 MAS code is always printed before execution.
• Begin with smaller MAX_PROBLEMS, fewer ROUNDS, or lower K_TRAJ in the shell scripts.
• --fresh vs resume: same --run-id resumes; add --fresh when you want a clean experiment directory.
• BrowseComp-Plus requires the BM25 index and official data pipeline — see dataset/BrowseComp-Plus/README.md.
• Cached trajectories and skills live under results/; add that path to .gitignore if you fork the repo.

---

7. BibTeX

@article{lin2026skill,
  title   = {Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems},
  author  = {Lin, Hehai and Yang, Qi and Qin, Chengwei},
  journal = {arXiv preprint arXiv:2606.18837},
  year    = {2026}
}